Hi,
I am struggling with some similar issues here and I am not sure what to do about it. Essentially I have a class that uses earthaccess and a set of gdal commands, then proceeds to obtain information about the granules. I can access the information from a search query and get a URL to a geotif file for download. But the vsicurl continous fails despite trying all of your above answers. I was hoping you could take a look at my code.
I have a .netrc file with a server address and my login details in my Google Cloud Virtual Machine home folder (Linux). I am running VS code on a windows laptop that then SSH's into the GC VM.
Here are parts and snippets of my code. The entire thing is long and contains some IP, so I cannot supply the whole thing:
class HLSNASA(xxxxx):
def __init__(self):
self.configure_gdal()
self.earth_access()
#earthaccess.login(persist=True)
@staticmethod
def earth_access():
earthaccess.login(persist=True)
@staticmethod
def configure_gdal():
# Set GDAL configuration options for HTTP handling and file extensions
#gdal.SetConfigOption("GDAL_HTTP_COOKIEFILE", "../cookies.txt") # I have tried this being commented in and out
#gdal.SetConfigOption("GDAL_HTTP_COOKIEJAR", "../cookies.txt")
gdal.SetConfigOption("GDAL_DISABLE_READDIR_ON_OPEN", "EMPTY_DIR")
gdal.SetConfigOption("CPL_VSIL_CURL_ALLOWED_EXTENSIONS", "TIF")
gdal.SetConfigOption("GDAL_HTTP_UNSAFESSL", "YES")
gdal.SetConfigOption('CPL_DEBUG', 'ON')
gdal.SetConfigOption('CPL_CURL_VERBOSE', 'ON')
[cut out part of the code: Here I basically create meta data based on file url, reformatting bounding box coordinate format etc]
hls_out = earthaccess.search_data(
short_name=["HLSL30", "HLSS30"],
bounding_box=bbox_coords,
temporal=(start_temporal, end_temporal),
count=count,
cloud_cover=cloud_cover_range,
)
[cut out code that parses this output]
which brings and output like this when I call the class and function in a jupyter notebook
Granules found: 1
2024-03-19T13:25:15-INFO-xxxxxx:iter_items:593 Found 1 items matching the search query.
and i can iterate over it and get feedback like this:
for item in item_iterator:
print(item)
break
https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.VZA.tif
Now this link I can open in my browser on my windows laptop and it immediatedly downloads the geotiff file.
But when I try to run this code in GC VM to iterate over the images I want to download (or in this case 1 as the search only returned 1):
def iter_images(
self,
start_date: str,
end_date: str,
geometry: BaseGeometry,
bands: list,
count: int = 500,
min_cloud: float = 0,
cloud_cover: float = 25,
access_type: str = "external",
clip_method: str = "window",
pixels: int = 1024,
sort: bool = True,
):
links = self.iter_items(
start_date=start_date,
end_date=end_date,
geometry=geometry,
count=count,
min_cloud=min_cloud,
cloud_cover=cloud_cover,
access_type=access_type,
sort=sort,
)
# Convert links to items with metadata
items_with_metadata = [(link, self._parse_granule_url(link)) for link in links]
# Filter for specified bands
filtered_items = [
item for item in items_with_metadata if item[1] and item[1]["band"] in bands
]
# Use concurrent futures to read and process images in parallel
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = [
executor.submit(self._process_image, item, geometry, clip_method, pixels)
for item in filtered_items
]
# Wait for all futures to complete and retrieve results
results = [future.result() for future in concurrent.futures.as_completed(futures)]
# Return an iterator over the results
return iter(results)
def _process_image(self, item_metadata, geometry, clip_method, pixels):
url, metadata = item_metadata
max_retries = 3
retry_delay = 5 # seconds to wait between retries
for attempt in range(max_retries):
try:
with rasterio.open(url) as src:
if clip_method:
clipper = ImageClipper.create_clipper(method=clip_method, pixels=pixels)
data, _, transform = clipper.clip_dataset(src=src, geometry=geometry)
else:
data = src.read()
transform = src.transform
# Successfully read the data, so break out of the retry loop
return data, transform, src.profile
except rasterio.errors.RasterioIOError as e:
print(f"Attempt {attempt+1}/{max_retries} failed with error: {e}. Retrying...")
time.sleep(retry_delay)
except Exception as e:
print(f"An unexpected error occurred: {e}. Aborting...")
break
else:
print(f"Failed to process {url} after {max_retries} retries. Check authentication.")
return None
I get this onslaught of errors
Granules found: 1
2024-03-19T13:25:18-INFO-skylens.stac_client:iter_items:593 Found 1 items matching the search query.
2024-03-19T13:25:25-INFO-rasterio._env:open:304 GDAL signalled an error: err_no=4, msg="`/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B04.tif' not recognized as a supported file format."
Attempt 1/3 failed with error: '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B04.tif' not recognized as a supported file format.. Retrying...
2024-03-19T13:25:25-INFO-rasterio._env:open:304 GDAL signalled an error: err_no=4, msg="`/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif' not recognized as a supported file format."
2024-03-19T13:25:25-INFO-rasterio._env:open:304 GDAL signalled an error: err_no=4, msg="`/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B02.tif' not recognized as a supported file format."
Attempt 1/3 failed with error: '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif' not recognized as a supported file format.. Retrying...
Attempt 1/3 failed with error: '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B02.tif' not recognized as a supported file format.. Retrying...
So I figured I'd try something a little more basic, so I opened a terminal, loaded in my environment that has gdal 3.7.3 and entered the followed command with debug on:
gdalinfo --config CPL_VSIL_CURL_ALLOWED_EXTENSIONS ".tif" --config CPL_DEBUG "ON" /vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif
HTTP: libcurl/8.5.0 OpenSSL/3.2.1 zlib/1.2.13 zstd/1.5.5 libssh2/1.11.0 nghttp2/1.58.0
HTTP: Using HTTP/2 for HTTPS when possible
VSICURL: Effective URL: https://d1nklfio7vscoe.cloudfront.net/head-2d2df3a34830d5223d1e9547cd713408/lp-prod-protected.s3.us-west-2.amazonaws.com/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif?Expires=1710858430&Signature=retRYPRFfuYR8YfaaDSXemEVUovyyXjmXGY4D0K0~9dBSwkkqE7Qnm~OgvD0WbqyKCdqACARmCLRITYv~~-SVGBH9xZDcxdFcWG8kMkWgXru4IBvvyJMnx2k7SBVNN3egRmAcfUpaOEaC0jvYiLHYd9yGT3NeNx4tNIwNSIIhFMk0uMSlVl0929BmtTw4PzYmzWcpqa2WHcMOld4mJAcpgLnqd-wigx5D5rIDNWvDEtI6EGrMU65IztahZ45NXWZ7GCjM15wRn2mJFmb3Ul1-maHdDUUV2IjppmHIMbH17d2GbGw7VpDut9FEe3wsa7q~HO67WU6VZfQbq2xrYL8kA__&Key-Pair-Id=K3SSHANJGSH7G9
VSICURL: Will use redirect URL for the next 3598 seconds
VSICURL: GetFileSize(https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif)=24650960 response_code=200
VSICURL: Using redirect URL as it looks to be still valid (3598 seconds left)
VSICURL: Downloading 0-16383 (https://d1nklfio7vscoe.cloudfront.net/head-2d2df3a34830d5223d1e9547cd713408/lp-prod-protected.s3.us-west-2.amazonaws.com/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif?Expires=1710858430&Signature=retRYPRFfuYR8YfaaDSXemEVUovyyXjmXGY4D0K0~9dBSwkkqE7Qnm~OgvD0WbqyKCdqACARmCLRITYv~~-SVGBH9xZDcxdFcWG8kMkWgXru4IBvvyJMnx2k7SBVNN3egRmAcfUpaOEaC0jvYiLHYd9yGT3NeNx4tNIwNSIIhFMk0uMSlVl0929BmtTw4PzYmzWcpqa2WHcMOld4mJAcpgLnqd-wigx5D5rIDNWvDEtI6EGrMU65IztahZ45NXWZ7GCjM15wRn2mJFmb3Ul1-maHdDUUV2IjppmHIMbH17d2GbGw7VpDut9FEe3wsa7q~HO67WU6VZfQbq2xrYL8kA__&Key-Pair-Id=K3SSHANJGSH7G9)...
VSICURL: Got response_code=403
VSICURL: Got an error with redirect URL. Retrying with original one
VSICURL: Downloading 0-16383 (https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif)...
VSICURL: DownloadRegion(https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif): response_code=302, msg=Maximum (10) redirects followed
VSICURL: Got response_code=302
VSICURL: Effective URL: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=FtSFfbOeuxDcdf4px-elGw&response_type=code&redirect_uri=https://data.lpdaac.earthdatacloud.nasa.gov/login&state=%2Flp-prod-protected%2FHLSS30.020%2FHLS.S30.T30UXD.2022026T111331.v2.0%2FHLS.S30.T30UXD.2022026T111331.v2.0.B03.tif&app_type=401
VSICURL: GetFileList(/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0)
ERROR 4: `/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif' not recognized as a supported file format.
gdalinfo failed - unable to open '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif'.
It seems to me it is an authorisation problem where GDAL is lacking the access to retrieve the images on my GC VM. The fact that I can click the link and download it without logging in is probably because its on my windows laptop. So my Google Cloud VM seems unable to grant GDAL the required permissions, despite the .netrc being in the home directory of my GC VM.
I understand I can define the access manually by defining the cookies.txt file. But I cannot find any information of what that file should specifically contain. I tried copy pasting the .netrc file content (1. server, 2. user, 3. pass) into a cookies.txt file and specifically hard linked its path to the GDAL_HTTP_COOKIEFILE and JAR. But that did not work either.
So I am at my wits end here. Any help is much appreciated
I am struggling with some similar issues here and I am not sure what to do about it. Essentially I have a class that uses earthaccess and a set of gdal commands, then proceeds to obtain information about the granules. I can access the information from a search query and get a URL to a geotif file for download. But the vsicurl continous fails despite trying all of your above answers. I was hoping you could take a look at my code.
I have a .netrc file with a server address and my login details in my Google Cloud Virtual Machine home folder (Linux). I am running VS code on a windows laptop that then SSH's into the GC VM.
Here are parts and snippets of my code. The entire thing is long and contains some IP, so I cannot supply the whole thing:
class HLSNASA(xxxxx):
def __init__(self):
self.configure_gdal()
self.earth_access()
#earthaccess.login(persist=True)
@staticmethod
def earth_access():
earthaccess.login(persist=True)
@staticmethod
def configure_gdal():
# Set GDAL configuration options for HTTP handling and file extensions
#gdal.SetConfigOption("GDAL_HTTP_COOKIEFILE", "../cookies.txt") # I have tried this being commented in and out
#gdal.SetConfigOption("GDAL_HTTP_COOKIEJAR", "../cookies.txt")
gdal.SetConfigOption("GDAL_DISABLE_READDIR_ON_OPEN", "EMPTY_DIR")
gdal.SetConfigOption("CPL_VSIL_CURL_ALLOWED_EXTENSIONS", "TIF")
gdal.SetConfigOption("GDAL_HTTP_UNSAFESSL", "YES")
gdal.SetConfigOption('CPL_DEBUG', 'ON')
gdal.SetConfigOption('CPL_CURL_VERBOSE', 'ON')
[cut out part of the code: Here I basically create meta data based on file url, reformatting bounding box coordinate format etc]
hls_out = earthaccess.search_data(
short_name=["HLSL30", "HLSS30"],
bounding_box=bbox_coords,
temporal=(start_temporal, end_temporal),
count=count,
cloud_cover=cloud_cover_range,
)
[cut out code that parses this output]
which brings and output like this when I call the class and function in a jupyter notebook
Granules found: 1
2024-03-19T13:25:15-INFO-xxxxxx:iter_items:593 Found 1 items matching the search query.
and i can iterate over it and get feedback like this:
for item in item_iterator:
print(item)
break
https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.VZA.tif
Now this link I can open in my browser on my windows laptop and it immediatedly downloads the geotiff file.
But when I try to run this code in GC VM to iterate over the images I want to download (or in this case 1 as the search only returned 1):
def iter_images(
self,
start_date: str,
end_date: str,
geometry: BaseGeometry,
bands: list,
count: int = 500,
min_cloud: float = 0,
cloud_cover: float = 25,
access_type: str = "external",
clip_method: str = "window",
pixels: int = 1024,
sort: bool = True,
):
links = self.iter_items(
start_date=start_date,
end_date=end_date,
geometry=geometry,
count=count,
min_cloud=min_cloud,
cloud_cover=cloud_cover,
access_type=access_type,
sort=sort,
)
# Convert links to items with metadata
items_with_metadata = [(link, self._parse_granule_url(link)) for link in links]
# Filter for specified bands
filtered_items = [
item for item in items_with_metadata if item[1] and item[1]["band"] in bands
]
# Use concurrent futures to read and process images in parallel
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = [
executor.submit(self._process_image, item, geometry, clip_method, pixels)
for item in filtered_items
]
# Wait for all futures to complete and retrieve results
results = [future.result() for future in concurrent.futures.as_completed(futures)]
# Return an iterator over the results
return iter(results)
def _process_image(self, item_metadata, geometry, clip_method, pixels):
url, metadata = item_metadata
max_retries = 3
retry_delay = 5 # seconds to wait between retries
for attempt in range(max_retries):
try:
with rasterio.open(url) as src:
if clip_method:
clipper = ImageClipper.create_clipper(method=clip_method, pixels=pixels)
data, _, transform = clipper.clip_dataset(src=src, geometry=geometry)
else:
data = src.read()
transform = src.transform
# Successfully read the data, so break out of the retry loop
return data, transform, src.profile
except rasterio.errors.RasterioIOError as e:
print(f"Attempt {attempt+1}/{max_retries} failed with error: {e}. Retrying...")
time.sleep(retry_delay)
except Exception as e:
print(f"An unexpected error occurred: {e}. Aborting...")
break
else:
print(f"Failed to process {url} after {max_retries} retries. Check authentication.")
return None
I get this onslaught of errors
Granules found: 1
2024-03-19T13:25:18-INFO-skylens.stac_client:iter_items:593 Found 1 items matching the search query.
2024-03-19T13:25:25-INFO-rasterio._env:open:304 GDAL signalled an error: err_no=4, msg="`/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B04.tif' not recognized as a supported file format."
Attempt 1/3 failed with error: '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B04.tif' not recognized as a supported file format.. Retrying...
2024-03-19T13:25:25-INFO-rasterio._env:open:304 GDAL signalled an error: err_no=4, msg="`/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif' not recognized as a supported file format."
2024-03-19T13:25:25-INFO-rasterio._env:open:304 GDAL signalled an error: err_no=4, msg="`/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B02.tif' not recognized as a supported file format."
Attempt 1/3 failed with error: '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif' not recognized as a supported file format.. Retrying...
Attempt 1/3 failed with error: '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B02.tif' not recognized as a supported file format.. Retrying...
So I figured I'd try something a little more basic, so I opened a terminal, loaded in my environment that has gdal 3.7.3 and entered the followed command with debug on:
gdalinfo --config CPL_VSIL_CURL_ALLOWED_EXTENSIONS ".tif" --config CPL_DEBUG "ON" /vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif
HTTP: libcurl/8.5.0 OpenSSL/3.2.1 zlib/1.2.13 zstd/1.5.5 libssh2/1.11.0 nghttp2/1.58.0
HTTP: Using HTTP/2 for HTTPS when possible
VSICURL: Effective URL: https://d1nklfio7vscoe.cloudfront.net/head-2d2df3a34830d5223d1e9547cd713408/lp-prod-protected.s3.us-west-2.amazonaws.com/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif?Expires=1710858430&Signature=retRYPRFfuYR8YfaaDSXemEVUovyyXjmXGY4D0K0~9dBSwkkqE7Qnm~OgvD0WbqyKCdqACARmCLRITYv~~-SVGBH9xZDcxdFcWG8kMkWgXru4IBvvyJMnx2k7SBVNN3egRmAcfUpaOEaC0jvYiLHYd9yGT3NeNx4tNIwNSIIhFMk0uMSlVl0929BmtTw4PzYmzWcpqa2WHcMOld4mJAcpgLnqd-wigx5D5rIDNWvDEtI6EGrMU65IztahZ45NXWZ7GCjM15wRn2mJFmb3Ul1-maHdDUUV2IjppmHIMbH17d2GbGw7VpDut9FEe3wsa7q~HO67WU6VZfQbq2xrYL8kA__&Key-Pair-Id=K3SSHANJGSH7G9
VSICURL: Will use redirect URL for the next 3598 seconds
VSICURL: GetFileSize(https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif)=24650960 response_code=200
VSICURL: Using redirect URL as it looks to be still valid (3598 seconds left)
VSICURL: Downloading 0-16383 (https://d1nklfio7vscoe.cloudfront.net/head-2d2df3a34830d5223d1e9547cd713408/lp-prod-protected.s3.us-west-2.amazonaws.com/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif?Expires=1710858430&Signature=retRYPRFfuYR8YfaaDSXemEVUovyyXjmXGY4D0K0~9dBSwkkqE7Qnm~OgvD0WbqyKCdqACARmCLRITYv~~-SVGBH9xZDcxdFcWG8kMkWgXru4IBvvyJMnx2k7SBVNN3egRmAcfUpaOEaC0jvYiLHYd9yGT3NeNx4tNIwNSIIhFMk0uMSlVl0929BmtTw4PzYmzWcpqa2WHcMOld4mJAcpgLnqd-wigx5D5rIDNWvDEtI6EGrMU65IztahZ45NXWZ7GCjM15wRn2mJFmb3Ul1-maHdDUUV2IjppmHIMbH17d2GbGw7VpDut9FEe3wsa7q~HO67WU6VZfQbq2xrYL8kA__&Key-Pair-Id=K3SSHANJGSH7G9)...
VSICURL: Got response_code=403
VSICURL: Got an error with redirect URL. Retrying with original one
VSICURL: Downloading 0-16383 (https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif)...
VSICURL: DownloadRegion(https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif): response_code=302, msg=Maximum (10) redirects followed
VSICURL: Got response_code=302
VSICURL: Effective URL: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=FtSFfbOeuxDcdf4px-elGw&response_type=code&redirect_uri=https://data.lpdaac.earthdatacloud.nasa.gov/login&state=%2Flp-prod-protected%2FHLSS30.020%2FHLS.S30.T30UXD.2022026T111331.v2.0%2FHLS.S30.T30UXD.2022026T111331.v2.0.B03.tif&app_type=401
VSICURL: GetFileList(/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0)
ERROR 4: `/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif' not recognized as a supported file format.
gdalinfo failed - unable to open '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T30UXD.2022026T111331.v2.0/HLS.S30.T30UXD.2022026T111331.v2.0.B03.tif'.
It seems to me it is an authorisation problem where GDAL is lacking the access to retrieve the images on my GC VM. The fact that I can click the link and download it without logging in is probably because its on my windows laptop. So my Google Cloud VM seems unable to grant GDAL the required permissions, despite the .netrc being in the home directory of my GC VM.
I understand I can define the access manually by defining the cookies.txt file. But I cannot find any information of what that file should specifically contain. I tried copy pasting the .netrc file content (1. server, 2. user, 3. pass) into a cookies.txt file and specifically hard linked its path to the GDAL_HTTP_COOKIEFILE and JAR. But that did not work either.
So I am at my wits end here. Any help is much appreciated
Statistics: Posted by kongstad_dk — Tue Mar 19, 2024 9:48 am America/New_York