Search Tutorial¶
1.1 Load packages¶
import sys
import pandas as pd
import ECAUGT
import time
import multiprocessing
import numpy as np
1.2 Connect to server¶
# set parameters
endpoint = "https://HCAd-Datasets.cn-beijing.ots.aliyuncs.com"
access_id = "LTAI5t7t216W9amUD1crMVos" #enter your id and keys
access_key = "ZJPlUbpLCij5qUPjbsU8GnQHm97IxJ"
instance_name = "HCAd-Datasets"
table_name = 'HCA_d'
# # setup client
ECAUGT.Setup_Client(endpoint, access_id, access_key, instance_name, table_name)
Connected to the server, find the table.
HCA_d
TableName: HCA_d
PrimaryKey: [('cid', 'INTEGER')]
Reserved read throughput: 0
Reserved write throughput: 0
Last increase throughput time: 1605795297
Last decrease throughput time: None
table options's time to live: -1
table options's max version: 1
table options's max_time_deviation: 86400
0
1.3 Build index¶
We should check if the index has been built.
ECAUGT.build_index()
index already exist.
2. Search cell with metadata condition¶
Conditions are presented in a structured string which is a combination of several logical expressions.
Each logical expression should be in the following forms:
field_name1 == value1, here '==' means equal
field_name2 <> value2, here '<>' means unequal
Three symbols are used for logical operation between expressions:
logical_expression1 && logical_expression2, here '&&' means AND operation
logical_expression1 || logical_expression2, here '||' means OR operation
！ logical_expression1, here '!' means not NOT operation
Brackets are allowed and the priorities of the logical operations are as common. The metadata condition string is also robust to the space character.
# get primary keys
rows_to_get = ECAUGT.search_metadata("organ == Lung && cell_type == T cell ")
14894 cells found
We found 14894 cells here, and the vairable rows_to_get is a list containing their primary keys.
3. Download data¶
We first download three columns of the queried cellls and return them in the DataFrame form. (The first column in the result is the primary keys)
For illustration, we only download the first 20 cells.
rows_to_get_2 = rows_to_get[0:20]
3.1 Download interested columns¶
# download data in pandas::DataFrame from
ECAUGT.get_columnsbycell_para(rows_to_get = rows_to_get_2, cols_to_get=['cl_name','hcad_name','cell_type'], col_filter=None, do_transfer = True, thread_num = multiprocessing.cpu_count()-1)
| cell_type | cl_name | uHAF_name | |
|---|---|---|---|
| cid | |||
| 2000932 | T cell | T cell | Lung-Connective tissue-T cell-CD3D CD8A |
| 2000962 | T cell | T cell | Lung-Connective tissue-T cell-CD3D IL32 |
| 2000971 | T cell | T cell | Lung-Connective tissue-T cell-CD3D IL32 |
| 2000978 | T cell | T cell | Lung-Connective tissue-T cell-CD3D CD8A |
| 2000987 | T cell | T cell | Lung-Connective tissue-T cell-CD3D IL32 |
| 2000994 | T cell | T cell | Lung-Connective tissue-T cell-CD3D IL32 |
| 2001027 | T cell | T cell | Lung-Connective tissue-T cell-CD3D IL32 |
| 2001030 | T cell | T cell | Lung-Connective tissue-T cell-CD3D CD8A |
| 2001031 | T cell | T cell | Lung-Connective tissue-T cell-CD3D IL32 |
| 2001032 | T cell | T cell | Lung-Connective tissue-T cell-CD3D CD8A |
| 2001038 | T cell | T cell | Lung-Connective tissue-T cell-CD3D IL32 |
| 2001050 | T cell | T cell | Lung-Connective tissue-T cell-CD3D CD8A |
| 2001059 | T cell | T cell | Lung-Connective tissue-T cell-CD3D CD8A |
| 2001065 | T cell | T cell | Lung-Connective tissue-T cell-CD3D IL32 |
| 2001086 | T cell | T cell | Lung-Connective tissue-T cell-CD3D IL32 |
| 2001091 | T cell | T cell | Lung-Connective tissue-T cell-CD3D IL32 |
| 2001099 | T cell | T cell | Lung-Connective tissue-T cell-CD3D IL32 |
| 2001105 | T cell | T cell | Lung-Connective tissue-T cell-CD3D IL32 |
| 2001106 | T cell | T cell | Lung-Connective tissue-T cell-CD3D CD8A |
| 2001112 | T cell | T cell | Lung-Connective tissue-T cell-CD3D CD8A |
Then we show how the result will look like when we don’t do transform.
# download data in list from
ECAUGT.get_columnsbycell_para(rows_to_get = rows_to_get_2, cols_to_get=['cl_name','hcad_name','cell_type'], col_filter=None, do_transfer = False, thread_num = multiprocessing.cpu_count()-1)
[[('cid', 2000932),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D CD8A')],
[('cid', 2000962),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D IL32')],
[('cid', 2000971),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D IL32')],
[('cid', 2000978),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D CD8A')],
[('cid', 2000987),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D IL32')],
[('cid', 2000994),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D IL32')],
[('cid', 2001027),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D IL32')],
[('cid', 2001030),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D CD8A')],
[('cid', 2001031),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D IL32')],
[('cid', 2001032),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D CD8A')],
[('cid', 2001038),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D IL32')],
[('cid', 2001050),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D CD8A')],
[('cid', 2001059),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D CD8A')],
[('cid', 2001065),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D IL32')],
[('cid', 2001086),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D IL32')],
[('cid', 2001091),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D IL32')],
[('cid', 2001099),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D IL32')],
[('cid', 2001105),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D IL32')],
[('cid', 2001106),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D CD8A')],
[('cid', 2001112),
('cell_type', 'T cell'),
('cl_name', 'T cell'),
('hcad_name', 'Lung-Connective tissue-T cell-CD3D CD8A')]]
3.2 Download all columns¶
We also compare the time comsumption between parallel and unparallel cell download processes for the first 20 cells, and find the parallel process only takes about 1/3 time.
# the parallel version
start_time = time.time()
result = ECAUGT.get_columnsbycell_para(rows_to_get = rows_to_get_2, cols_to_get=None, col_filter=None, do_transfer = False, thread_num = multiprocessing.cpu_count()-1)
time.time()-start_time
5.3618810176849365
# the unparallel version
start_time = time.time()
result = ECAUGT.get_columnsbycell(rows_to_get = rows_to_get_2, cols_to_get=None,col_filter=None,do_transfer = False)
time.time()-start_time
29.58238458633423
4. Search cell with both metadata condition and gene condition¶
Now we show how to add gene conditions when downloading cells. Here we download some genes of the queried cells and select the cells whose expression level on PTPRC is larger than 0.1 and experssion level on CD3D is no less than 0.1
# add col_filter on gene
gene_condition = ECAUGT.set_gene_condition("PTPRC > 0.1 && CD3D>=0.1")
4.1 Download some of the columns¶
df_result = ECAUGT.get_columnsbycell_para(rows_to_get = rows_to_get, cols_to_get=['CD3D','PTPRC','donor_id','hcad_name'], col_filter=gene_condition, do_transfer = True, thread_num = multiprocessing.cpu_count()-1)
df_result
| CD3D | PTPRC | donor_id | uHAF_name | |
|---|---|---|---|---|
| cid | ||||
| 2000962 | 2.598072 | 2.229140 | 343B | Lung-Connective tissue-T cell-CD3D IL32 |
| 2000987 | 2.138744 | 1.790511 | 343B | Lung-Connective tissue-T cell-CD3D IL32 |
| 2000994 | 3.055748 | 2.269341 | 343B | Lung-Connective tissue-T cell-CD3D IL32 |
| 2001099 | 2.682663 | 1.864017 | 343B | Lung-Connective tissue-T cell-CD3D IL32 |
| 2001112 | 2.417518 | 1.482966 | 343B | Lung-Connective tissue-T cell-CD3D CD8A |
| ... | ... | ... | ... | ... |
| 2115395 | 2.729593 | 2.729593 | FetalLung1_12W | Lung-Connective tissue-T cell-CD3D CD1C |
| 2115433 | 3.851911 | 3.179780 | FetalLung1_12W | Lung-Connective tissue-T cell-CD3D CD1C |
| 2115441 | 3.591656 | 2.546684 | FetalLung1_12W | Lung-Connective tissue-T cell-CD3D CD1C |
| 2115483 | 3.181991 | 3.181991 | FetalLung1_12W | Lung-Connective tissue-T cell-CD3D CD1C |
| 2115498 | 3.124087 | 3.124087 | FetalLung1_12W | Lung-Connective tissue-T cell-CD3D CD1C |
7403 rows × 4 columns
We found that 7403 cells among the 14889 queried cells has expression levels that satisfy PTPRC > 0.1 && CD3D>=0.1. Then we can download some columns of these cells with the parameter cols_to get and the genes involved in the condition must be included in the
4.2 Download all columns of these cells¶
We can get all expression levels and metadatas of these cells by setting the parameter cols_to_get as None
df_result = ECAUGT.get_columnsbycell_para(rows_to_get = rows_to_get, cols_to_get=None, col_filter=gene_condition, do_transfer = True, thread_num = multiprocessing.cpu_count()-1)