GHG Europe Database

 

Data submission - General Information

Introduction

This document describes how to prepare and upload data to the database. Please read it carefully and if some step is not clear or for any further information send an email to the database staff at database@unitus.it.

Please note that data can be upload only following the method explained here and through the upload service web page available in the PI area. Data sent by email will not be considered.

At the end of this page there is a short step-by-step instruction for data upload.

File upload

Files containing the data can be submitted only using the “Upload data” service in the PI Area available in the top bar menu.

Login

In order to access the PI Area and the Upload Service you have to provide your username and password; an email containing the personal login details has been sent to the site PI and collaborators indicated by the PI.

If you don't have an account valid to submit data please ask the site PI to add you to the Collaborators list using the tools available in the PI Area. If you do not remember the username or password use the “Forgotten password?” tool in the login page, enter your email address and a message containing your login details will be sent to your email address.

Upload process

After login you will be asked to provide a number of information in addition to the file to upload. Use the “data info” field to send information about the additional variables you sent or comment about a new version of your data. You can send the data as excel file or ASCII file, see the Data format section for details.

IMPORTANT: Do not write non standard ASCII characters in either the file and its information (data info) posted in the page otherwise an error message will appear during the upload process.

Click on “Submit” button, a dialog box will display the upload progress. Please note that it is not possible to submit compressed files.

The received files are evaluated and if they are formally corrected, they are imported and processed.

New versions of the data

When a variable already present in the database is resubmitted, will be imported and overwritten in the system, creating a new version of the products. The new version will be described through the information reported by the PI in the “data info” field during the submission step ("Upload data" page).

Data format for continuous measurements

A continuous dataset is a set of measurements that are done at regular intervals of time, generally daily or more frequent, for a certain period. The meteorological and fluxes measurements are one example but also automatic chambers measurements (e.g. every three hours) or daily precipitations are also an example of continuous measurements. To submit these data it is possible to use the format reported here below. The format of the files submitted is very important; in facts, due to the high volume of data files coming to database, the importing process is completely automated and files not fitting the given format are refused (an error message is sent to file sender).

General rules

The first two fields of the file must be always date and time. Time indicate the END of the averaging period and midnight must be "0:00” of the following day

Example:

08/10/2009 23:30    averaging period: 8/10/2009 23:00 - 23:30

09/10/2009 0:00      averaging period: 8/10/2009 23:30 - 24:00 (0:00)

09/10/2009 0:30      averaging period: 9/10/2009 0:00 - 0:30


Time convention: report the local time without “Daylight Saving Time”.

Missing data must be indicated as -9999.

Text format

Text ASCII files must contain the variables codes in their first line. Variable names have to be chosen according with the list rules reported in the section below. First two columns have to report date and time according with the general instructions above. The comma separator is advised.


Example:

date,time,CO2,H2O,ZL,Fc

dd/mm/yyyy,hh:mm,umol/mol,mmol/mol,,umol m-2 s-1

07/10/2002,06:00,375.0026343,13.81902137,0.075133,2.225711711

07/10/2002,06:30,375.6178651,13.81904135,0.067499,1.611090395

07/10/2002,07:00,375.1484745,13.77998531,0.032477,1.11762877

07/10/2002,07:30,374.0334503,13.73454349,0.024741,0.236125726


An example of the ASCII template is provided at the end of this page.

Excel format

Data must be placed in the first sheet of the file (you can check it on the bottom left corner of excel) and the first row must be filled with the variables codes according with the naming conventions explained below.

Be sure that only numeric data are in the cells (no characters, no DIV/0…), and fill the empty cells with -9999 because data series are read until the first empty cell in the date column is found.

An example of the Excel template is provided at the end of this page.

Rules for variables names

To submit the data it is fundamental to use the correct variables names and units as reported in the Variables codes page. However, in addition to the variable name it is requested to use a 3 figures code which identifies univocally the location within the site where measurements are taken. The importance of this code is related to the possibility to associate to each variable a set of metadata like the method or instrument used, the measurement depth/height, the last calibration of the sensor etc.

Generally, the name of the variable has to be indicated as:


VAR_x1_x2_x3


VAR= variable name (refer to the list of input variables)

The first figure stands for the position of the sensors on a 2D space. For instance, if soil temperature is measured at 3 different locations, these can be numbered as 1, 2, 3.


 

Ts_1_x_x

 

Ts_2_x_x

 

Ts_3_x_x

location 1

 

location 2

 

location 3


The second figure represents the vertical level (height above the ground/depth of the soil) of the sensors. Supposing to measure soil temperature at 3 different depths along a vertical profile at each location, then each variable can be identified as follows:


 

Ts_1_1_x

 

Ts_2_1_x

 

Ts_3_1_x

 

depth 1

Ts_1_2_x

 

Ts_2_2_x

 

Ts_3_2_x

 

depth 2

Ts_1_3_x

 

Ts_2_3_x

 

Ts_3_3_x

 

depth 3

location 1

 

location 2

 

location 3

 


Numeration increases moving downwards along a vertical profile but same number at two different locations can indicate different height/depth, that is specified in the metadata. So for example the deeper soil temperature measurement point in two different profiles of 4 sensors each will be indicated as Ts_1_4_1 and Ts_2_4_1 even if the first is at 80cm and the second 120cm below ground.

The third figure identifies the presence of replicated measurements in the same location. For instance, given 2 replicates of soil temperature measurements at position 1 and depth 1 (i.e. for sensors comparison), these would be indicated as:


 

Ts_1_1_1   Ts_1_1_2

Another example where the third figure should be used is for measurements at tree level like sap flow sensors. In this case the first figure indicate the tree (again specified in the metadata), the second the measurement height if you have for example sap flow measurements on the same tree at different height and the third can be used to identify different sensors at the same height, commonly used in sap flow measurements.

Remember: All variables to be submitted must be characterized by the 3 figure code and in case of a variable being measured with a single sensor in a given point in space, the three figures of the code clearly equals 1.

New variables

We tried to consider all the common variables collected in the network also on the basis of what has been collected in previous projects. However new measurements are continuously added and it is possible that a variable to be submitted is not present in the “Variables codes”. In this case please contact the Databse Staff at the address database@unitus.it and we will think together to the best solution. In any case do not send variables with names not consistent with those listed because these would be rejected by the system.

IMPORTANT: Additional information regarding the variables (height, depth) must be included in the metadata and not in the data file neither in the variable name. All additional information included in the data file are lost during the upload process.


Not continuous measurements, ancillary information and metadata

The not continuous data are variables measured sporadically, at not regular intervals of time or at low time resolution (e.g. monthly) that make the use of the template here explained easier for the data submitter. The information about the ecosystem and the methodologies are crucial to correctly interpret and use the fluxes measurements. For this reason all the PIs are invited to submit the relevant ancillary data and metadata.

General information

The template to submit ancillary data is the BADM (Biological, Ancillary, Disturbances and Metadata template) available at the end of this page. The BADM is an international standard used also in Ameriflux and FLUXNET and the format is directly importable in the database structure

For this reason the ancillary variables and the metadata must be submitted using the BADM. There are variables that can be submitted using both the continuous data format or the BADM, the amount of data and the time resolution should be considered when the format is decided (e.g, automatic chambers can be submitted as continuous data but the variable code must be the same used in the BADM).

IMPORTANT: the final BADM versions are still under finalization and are released as separated templates as soon as finally agreed internationally. Currently the final BADM are available to send data about the general characteristics of the site and the metadata about the instruments and link between instruments and variables submitted. The final approved templates are available here:

  1. BADM for the general site information and example

  2. BADM for the instruments description and instructions

  3. BADM for the instruments operation and variables mapping and instructions

For all the other data not included in the final template it is possible to use a temporary template available here. The rules to fill in the template and the structure is similar to the final version and the data submitted with this template will be automatically translated in the final version as soon as approved.

Instructions for completing BADM templates

Use column D and higher in the first worksheet to enter your information. Only use the columns past D when you have more than one entry that you need to put in a row. If you have nothing to enter in a row, leave it blank.

Do not enter information in shaded columns (columns A, B and C, first raw).

Variables with units of LIST(xxx)—see second worksheet in each template—should use only the choices listed. For the Temporary Template the options are provided in the pick-up list

Enter all values in the units requested. For numeric entries, use only numbers (no letters or symbols).

Fill in the submitter’s name—whoever is filling in the template (likely you).

See Explanations, the third worksheet, for variable groups that must be reported together:

  1. Column A lists the variable groups.

  2. Column C has a “p” if a variable is the primary variable for a group of variables.

  3. Column D lists an “r” if a variable is a required qualifier for a primary variable. For example, if H2O is entered for FLUX_MEASUREMENTS_VARIABLE on the Site-General-Info Template, the FLUX_MEASUREMENTS_DATE_START is required.

Don’t forget that…

  1. All the files must be submitted using the data upload service web page exclusively: files sent as mail attachment will not be considered.

  2. Do not write non standard ASCII characters in the file information field (data info) during the upload process.

  3. When continuous data are submitted, the first two fields must always be date and time; timestamp indicates the end of the averaging period and midnight (average of measurements from 23:30 to midnight) must be reported as 0:00 of the following day.

  4. Continuous data Excel format: data must be placed in the first sheet of the file, other sheets are not considered in the data import process.

  5. Continuous data text format: text files must contain the variable codes in the first line.

  6. Additional information regarding the variables (height, deepness, instruments,…) must be included in the BADM template and not in the data file neither in the variable name. All additional information included in the data file are lost during the automatic upload process.

  7. If a variable needs to be cancelled because errors have been found, it is possible to send again only the variable corrected that will overwrite the one imported (attention: the -9999 value also will overwrite the old value).

  8. Remember that for data processing, these variables are essential: FC, SC or CO2, SW_IN or PPFD, TA, TS, USTAR or TAU, RH. Very important are also SWC and Precipitation.

  9. Ancillary information, metadata and not continuous measurements must be submitted using the BADM template. Also maps and photos can be uploaded using the upload file tool in the PI Area

 

Templates available