7  Converting CSV data to Raster

This article is based on a question in QGIS community group. I have got a csv data which has evenly placed points in x and y direction. We can convert such data to raster in various ways. In the past I used FORTRAN to convert the data to raster. FORTRAN is faster and Python is easier. So, in this article I will choose the easier one.

Following is the format of data set

The xyz data
V1 V2 V3
696179.8 8649501 -19.231
696180.0 8649501 -19.242
696180.2 8649501 -19.235
696180.5 8649501 -19.229
696180.8 8649501 -19.215
696181.0 8649501 -19.230
696181.2 8649501 -19.232
696181.5 8649501 -19.228
696181.8 8649501 -19.216
696182.0 8649501 -19.207

For conversion, first of all we need to read the data into matrix form and then convert it to raster. I will list the approaches below.

7.1 Approach-1: Create an empty raster and fill with the data

  • Read the file into pandas dataframe or numpy matrix.
  • Find the minimum and maximum x and y values.
  • Find the minimum difference between to consecutive x and y values. This will be the cell-size of the raster.
  • Using PyQGIS create raster data with the above parameters (Extent, cell-size, CRS etc) and all null (NoData) values
  • Set the values of raster cells from the dataframe or the matrix

7.2 Approach-2: Read the data into matrix, save as ESRI ascii raster format then convert to tif

  • Read the file into pandas dataframe or numpy matrix.
  • Find the minimum and maximum x and y values.
  • Find the minimum difference between to consecutive x and y values. This will be the cell-size of the raster.
  • Find the number of rows and columns (say: numrows, numcols) of data by dividing the minimum and maximum values of x and y by the cell-size
  • Create a numpy/pandas matrix of size \(numrows \times numcols\) and values equal to NoData (-9999.0)
  • Fill other values from the matrix
  • Write header and matrix values to ascii file
  • This link shows the format of ascii file as shown below.
ncols 480
nrows 450
xllcorner 378923
yllcorner 4072345
cellsize 30
nodata_value -32768
43 2 45 7 3 56 2 5 23 65 34 6 32 54 57 34 2 2 54 6 
35 45 65 34 2 6 78 4 2 6 89 3 2 7 45 23 5 8 4 1 62 ...

7.3 Python code

Following code will convert the csv data to raster in ascii format. The code can be converted to function and multiple files can be converted to raster at once. I have kept the option open for the enthusiasts.

import pandas as pd
import numpy as np

# Install pandas and numpy if not done yet
# Create dataframe: Check the input file name, header and separator in your file
df = pd.read_csv('../Dataset.csv', header=None, sep=' ')
# Set the names of columns as x, y, z
df.columns = ['x', 'y', 'z']
# Get minimum and maximum values of x and y to find the number of rows and columns
dfmin = df.min()
dfmax = df.max()
# Number of rows = (ymax-ymin)/cellsize [here cellsize is 0.25]
numcols = int((dfmax['x'] - dfmin['x']) * 4) + 1
numrows = int((dfmax['y'] - dfmin['y']) * 4) + 1

print(numrows, numcols)

# Create a no_data matrix of size and shape of
mat = np.full((numrows, numcols), -9999.0)
# Replace the available values from the dataframe to the Numpy Matrix 'mat'
for index, row in df.iterrows():
    i = int((row['x'] - dfmin['x']) * 4) # 1/cellsize = 4
    j = int((row['y'] - dfmin['y']) * 4)
    mat[numrows-j-1][i] = row['z']

# We create an ascii raster file which is a text format
# For detailed explanation check here:
# https://desktop.arcgis.com/en/arcmap/latest/manage-data/raster-and-images/esri-ascii-raster-format.htm
xll = dfmin['x'] - 0.125
yll = dfmin['y'] - 0.125
header = f'''ncols        {numcols}
nrows        {numrows}
xllcorner    {xll}
yllcorner    {yll}
cellsize     0.250
'''
print('writing to file')
# Write the header
with open('rast.asc', 'w') as f:
    f.write(header)

# Add the matrix
with open("rast.asc", "a") as f:
    np.savetxt(f, mat, delimiter=" ", fmt = '%8.3f')
print('completed')

7.4 Running the python code and converting to TIF file

The above code can be run by open OSGeo4W shell. After running the above code, it is possible to convert to Tiff using gdal_translate as follows.

# CD to the appropriate path
# Run python script
python abc.py
# Run command to convert to GeoTIFF (Put the CRS appropriately)
gdal_translate rast.asc rast01.tif -a_srs EPSG:32645

Now the Raster data can be opened in QGIS. If the file size is big, we need to break the files into multiple files and create tiff files one by one. After that we need to merge the tiff files. While merging the tiff files, it is necessary to choose the maximum value option so that the NoData values will be replaced by actual values when two or more rasters are overlapping.