Uploading
An in depth overview of how to upload Samples, Protocols, and Data Files to NExtSEEK
Where to upload
NExtSEEK lets you register samples and attach data through the web app — one metadata sheet at a time, or in bulk. There are two upload pages:
Sample metadata —
/seek/samples/upload/Data and protocol files —
/seek/data/upload/
The first registers samples and their metadata from a spreadsheet; the second attaches data files and protocols (SOPs) to work you have already registered. This page covers both, plus how to check a sheet before you commit it.
Sheet formats
Samples are tabular metadata, uploaded as an Excel workbook. NExtSEEK accepts your metadata in either of two shapes and detects which one you used automatically from the sheet names — you do not pick a format.
4-sheet workbook
Meant for authoring by hand. The workbook has four named sheets — Instructions, Samples, Ontology, and Assay — and describes a single sample type.
Instructions: This sheet contains all of the information required to add the sample to the database. There are four required columns: Field, Database Field, Field Type, and Ontology.
Field = An identical match to the headers of the Samples sheet. The headers/column names do NOT need to be the database name of the attribute.
Database Field = Formatted as SAMPLETYPE::AttributeName — e.g. TIS::Type or D.SEQ::Name. The attribute name here MUST exactly match the database field name. This maps the value in the Samples sheet to the correct attribute.
Field Type = Text, Number, Date, or Controlled Ontology.
Ontology = If Field Type is Controlled Ontology, the name of the ontology (a column header in the Ontology sheet).
Samples: This is the table of metadata, where each row is a sample and each column is an attribute for that sample. The column headers (row 1) are the attribute names, and must identically match the Field column (transposed) of the Instructions sheet.
For samples, Name and File_Primary Data must be unique (per sample type). There cannot be two samples in the database with the same Name or File_PrimaryData (per sample type).
Ontology: This sheet contains ontologies (sets of controlled-vocabulary terms) that can be used to control the values of an attribute. For an ontology to be enforced, the Field Type on the Instructions sheet for that attribute must be set to Controlled Ontology, and the name of the ontology (its header in the Ontology sheet) must be set as the Ontology of that attribute on the Instructions sheet. See the image below for clarification.
Assay: This sheet determines which assay(s) the uploaded samples should be associated with. The required columns are: SampleType, AssayType, Assay, Direction.
Attached is an example sample sheet, with notes/annotations as described above.
Flat workbook
A single Samples sheet whose columns are uid, sampletype, json_metadata, and assay_ids, plus the optional columns project_id, study_title, and assay_titles. Here the metadata travels as a JSON blob in the json_metadata cell, so one flat sheet can carry many sample types at once.
In both formats, leaving a row's uid blank tells NExtSEEK to auto-generate an identifier for that sample. Fill uid in only when you are pointing at a sample that already exists. To load several sample types together, use one flat sheet, or submit several single-type 4-sheet files in the same upload (they are merged into one job).
Sample sheets, assay sheets, and update sheets
These are conventions for how the formats above get used:
Sample / assay sheets use the 4-sheet format and register samples for the first time. Leave the UID column blank on the first upload — NExtSEEK generates the UIDs. When the job finishes, download the summary and paste the generated UIDs back into your sheet so you can reference those samples later.
Historically an assay sheet bundled several sample types and a sample sheet a single type. A single 4-sheet workbook now describes exactly one sample type, so to register several types together, submit one 4-sheet file per type in the same upload, or use a flat sheet.
Update sheets change samples that already exist. Include the existing UIDs and supply only the fields you want to change — see Updating existing records.
Below is an example of an update sheet. (A sample-sheet example is linked above, SampleSheetFormatting_Template_240824.xlsx.)
Validate before you insert
Before committing anything, validate a sheet and get back exactly what would go wrong — without writing a single record. On the upload page, choose your sheet and a project and click Validate Samples; programmatically the same check is:
The validator runs the same parsing, naming, and structure checks as a real upload but stops short of the database. It returns structured errors grouped by the distinct problem — each group tells you the error type, a message, and which rows and UIDs are affected — so a sheet where fifty rows share one mistake shows up as a single issue rather than fifty. Nothing is inserted and no project links are changed.
Choose your prepared sheet, select a project, and click validate.
The validator checks:
The workbook format is recognized (4-sheet vs flat), detected automatically from the sheet names.
For a 4-sheet file: the four sheets are present; Instructions has its required columns (Field, Database Field, Field Type, Ontology); Assay has its required columns (SampleType, AssayType, Assay, Direction); and every Samples column header is declared in Instructions → Field.
Each Database Field maps to a real attribute on that sample type. (In the screenshot the type CEL has no attribute "Protocols" — it should be "Protocol".)
Controlled-ontology values fall within their declared vocabulary.
Optionally, a name check flags rows whose name/identity already exists (a duplicate, or a row that will update an existing sample), and a dependency check flags cycles in parent/child references.
Not every finding blocks the upload. Structural problems — a missing sheet or column, a bad format — will stop it. A softer issue is reported as a warning: for example, a Samples column that isn't declared in Instructions → Field is dropped, so that sample uploads without that attribute rather than failing. It is good practice to validate before every upload.
How to upload
Prepare your assay, sample, or update sheet (or a flat sheet).
Run it through Validate Samples and fix anything reported.
Drop your sheet(s) into the upload box and select a project. If you are an admin, choose which lab and creator you are uploading for; otherwise leave the defaults and it uploads as you.
Leave "Update existing? (otherwise skipped)" checked if any rows carry UIDs you intend to update; see Updating existing records. Click Upload Samples. The upload runs as a background job, so a large sheet does not tie up your browser — you can follow its progress on the page while it runs.
When the job finishes, download the summary. It reports the outcome for every row, including any UIDs that were auto-generated — paste those back into your sheet for future updates. You can also confirm on the Search page (for example, search today's date in YYMMDD format — so 8/23/24 = 240823 — and watch the sample count climb).
Quality checks: spot-check a few samples, confirm the expected number of samples were created, and confirm all of their attributes are present.
What happens when you insert
A real upload runs as a background job through an ordered pipeline. Each stage hands off to the next:
Convert — detect the format, merge multiple files, and run the ontology checks.
Name check — compare against existing samples to catch duplicates and to line up updates.
UID generation — assign identifiers to any rows that left
uidblank.Graph and levels — read the parent/child references between your samples, detect any dependency cycles, and order the samples so parents are inserted before their children.
Prefetch and transform — resolve sample-type attributes and assay links, and shape each row into an insertable record.
Insert — write the samples to the database, level by level.
Graph sync — mirror the new samples and their relationships into the sample graph (Neo4j), when the operator has enabled it.
Report — produce the per-row summary you can download.
The job checkpoints its progress as it inserts, so if it is interrupted it can resume where it left off rather than start over.
Updating existing records
To change samples that already exist, re-upload a sheet whose rows carry the existing UIDs, and make sure "Update existing? (otherwise skipped)" is checked on the upload form. With that option off, a row whose UID is already in the database is skipped as a duplicate rather than changed.
When update is on, NExtSEEK deep-merges the new metadata into the stored record: fields you include are overwritten with the new values, and fields you leave out are preserved. That makes a re-upload a safe patch — you supply only the fields you want to change, not the whole record.
One exception: assay links are replaced, not merged. A sample ends up linked to exactly the assays listed for it on the sheet, so any assay you omit is unlinked. If you are updating metadata and want to keep a sample's existing assay associations, list them again.
Uploading protocols and data files
To upload protocols and data files, head to the Protocol / Data File Uploading page.
Select whether the file(s) you are uploading are Data Files or Protocols.
If you are an admin, select which lab/user you are uploading for. If not, leave the default.
Place the files into the "File Dropzone" and click submit. Wait for the data files/protocols to upload.
The UID generated in the bottom table is the UID used to reference that data file or protocol. Data file UIDs are the matching sample's UID followed by the file name (
SampleUID_FileName). Protocol UIDs areP.LAB-YYMMDD-V<version>_FileName.
For protocols, following the procedure above is sufficient to upload.
Data files require a sample with a File_PrimaryData equal to the name of the file you are uploading (so the data file's UID / Link_PrimaryData is matched to the corresponding sample automatically). If there is no D. sample that matches your data file name, you can register a D.FILE sample to get the system to accept it — this is useful when the file is not a primary file but a supplementary one, such as a FASTQC.html. A D.FILE template is attached below.
Last updated