The Wayback Machine - https://web.archive.org/web/20250520064431/https://github.com/localstack/localstack/issues/4488
Skip to content

Issue when trying to create lambda function that depends on layers (PRO) #4488

New issue
New issue
Closed
Closed
Issue when trying to create lambda function that depends on layers (PRO)#4488
Assignees
whummer
Labels
aws:lambdaAWS Lambdastatus: triage neededRequires evaluation by maintainerstype: bugBug report
@mbenedettini

Description

@mbenedettini
mbenedettini
opened on Aug 25, 2021

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

When trying to create a lambda function I can see in the logs that there's an error about a module not being found (uriutils) that should be provided by the TextractionPython layer:

localstack_main  | 2021-08-25T15:08:25:DEBUG:localstack_ext.services.awslambda.lambda_extended: Copy function code layer arn:aws:lambda:us-west-1:000000000000:layer:poppler:1
localstack_main  | 2021-08-25T15:08:25:DEBUG:localstack.utils.run: Executing command: cd /tmp/localstack/lambda-with-layers.af8b42c5/__layers__; unzip -o -q /tmp/localstack/zipfile.7b20f340/original_lambda_archive.zip
localstack_main  | 2021-08-25T15:08:25:DEBUG:localstack_ext.services.awslambda.lambda_extended: Copy function code layer arn:aws:lambda:us-west-1:000000000000:layer:TextractionPython:1
localstack_main  | 2021-08-25T15:08:25:DEBUG:localstack.utils.run: Executing command: cd /tmp/localstack/lambda-with-layers.af8b42c5/__layers__; unzip -o -q /tmp/localstack/zipfile.75bde761/original_lambda_archive.zip
localstack_main  | 2021-08-25T15:08:26:DEBUG:localstack.utils.run: Executing command: cd /tmp/localstack/lambda-with-layers.af8b42c5; unzip -o -q /tmp/localstack/zipfile.2579c2d7/original_lambda_archive.zip
localstack_main  | 2021-08-25T15:08:26:DEBUG:localstack.utils.run: Executing command: cd /tmp/localstack/lambda-with-layers.af8b42c5; unzip -o -q /tmp/localstack/lambda-with-layers.af8b42c5/original_lambda_archive.zip
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_api.py", line 808, in _do_exec_lambda_code
localstack_main  |     handler_module = load_source(lambda_id, lambda_file)
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_api.py", line 780, in load_source
localstack_main  |     return importlib.machinery.SourceFileLoader(name, file).load_module()
localstack_main  |   File "<frozen importlib._bootstrap_external>", line 407, in _check_name_wrapper
localstack_main  |   File "<frozen importlib._bootstrap_external>", line 907, in load_module
localstack_main  |   File "<frozen importlib._bootstrap_external>", line 732, in load_module
localstack_main  |   File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
localstack_main  |   File "<frozen importlib._bootstrap>", line 696, in _load
localstack_main  |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
localstack_main  |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
localstack_main  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
localstack_main  |   File "/tmp/localstack/lambda_script_l_ef732d3a.py", line 8, in <module>
localstack_main  |     from uriutils import uri_read, uri_exists, uri_dump
localstack_main  | ModuleNotFoundError: No module named 'uriutils'
localstack_main  |
localstack_main  | 2021-08-25T15:08:27:DEBUG:localstack.services.awslambda.lambda_api: Unable to get handler function from lambda code.
localstack_main  | 2021-08-25T15:08:27:ERROR:lambda_api: Exception on /2015-03-31/functions [POST]
localstack_main  | Traceback (most recent call last):
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_api.py", line 1055, in do_set_function_code
localstack_main  |     lambda_env=lambda_environment,
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_api.py", line 834, in exec_lambda_code
localstack_main  |     return _do_exec_lambda_code()
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_api.py", line 822, in _do_exec_lambda_code
localstack_main  |     raise e
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_api.py", line 808, in _do_exec_lambda_code
localstack_main  |     handler_module = load_source(lambda_id, lambda_file)
localstack_main  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_api.py", line 780, in load_source
localstack_main  |     return importlib.machinery.SourceFileLoader(name, file).load_module()
localstack_main  |   File "<frozen importlib._bootstrap_external>", line 407, in _check_name_wrapper
localstack_main  |   File "<frozen importlib._bootstrap_external>", line 907, in load_module
localstack_main  |   File "<frozen importlib._bootstrap_external>", line 732, in load_module
localstack_main  |   File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
localstack_main  |   File "<frozen importlib._bootstrap>", line 696, in _load
localstack_main  |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
localstack_main  |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
localstack_main  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
localstack_main  |   File "/tmp/localstack/lambda_script_l_ef732d3a.py", line 8, in <module>
localstack_main  |     from uriutils import uri_read, uri_exists, uri_dump
localstack_main  | ModuleNotFoundError: No module named 'uriutils'

Expected Behavior

I should be able to create the function

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

Relevant part from my compose file:

lambda:
    image: localstack/localstack:0.12.17
    expose:
      - 80
    container_name: localstack_main
    hostname: lambda.localhost
    privileged: true
    networks:
      glide:
        aliases:
          - lambda.localhost
    environment:
      - VIRTUAL_HOST=lambda.localhost
      - VIRTUAL_PORT=80
      - HOSTNAME_EXTERNAL=lambda
      - LOCALSTACK_SERVICES=lambda
      - EDGE_PORT=80
      - LAMBDA_PORT_EXTERNAL=80
      - LOCALSTACK_API_KEY=........
      - DEBUG=1
    depends_on:
      - nginx
      - s3

Command used to create first layer (source: https://github.com/jeylabs/aws-lambda-poppler-layer):

aws --endpoint-url=http://lambda.localhost:80 lambda publish-layer-version --layer-name poppler --license-info "MIT" --zip-file fileb:///tmp/poppler.zip --compatible-runtimes python3.6 python3.7 python3.8

Command used to create the second layer:

aws --endpoint-url=http://lambda.localhost:80 lambda publish-layer-version --layer-name TextractionPython --license-info "MIT" --zip-file fileb://./textraction_python.zip --compatible-runtimes python3.8

I can't provide the full contents of that zip but it does include the package that the lambda is complaining about (uriutils) and I think the file structure is right:

 unzip -vl textraction_python.zip | grep uriutils
       0  Stored        0   0% 08-24-2021 15:53 00000000  python/lib/python3.8/site-packages/uriutils/
    9316  Defl:N     2641  72% 08-24-2021 15:53 3d20a895  python/lib/python3.8/site-packages/uriutils/uriutils.py
      24  Stored       24   0% 08-24-2021 15:53 90481da0  python/lib/python3.8/site-packages/uriutils/__init__.py
       0  Stored        0   0% 08-24-2021 15:53 00000000  python/lib/python3.8/site-packages/uriutils/__pycache__/
     172  Defl:N      139  19% 08-24-2021 15:53 4293b49e  python/lib/python3.8/site-packages/uriutils/__pycache__/__init__.cpython-38.pyc
    9044  Defl:N     4033  55% 08-24-2021 15:53 c8d4f931  python/lib/python3.8/site-packages/uriutils/__pycache__/uriutils.cpython-38.pyc
   21285  Defl:N     7782  63% 08-24-2021 15:53 d14a92d9  python/lib/python3.8/site-packages/uriutils/__pycache__/storages.cpython-38.pyc
   20017  Defl:N     4284  79% 08-24-2021 15:53 b0fe31d0  python/lib/python3.8/site-packages/uriutils/storages.py
       0  Stored        0   0% 08-24-2021 15:53 00000000  python/lib/python3.8/site-packages/uriutils-0.1.16.dist-info/
       1  Stored        1   0% 08-24-2021 15:53 32d70693  python/lib/python3.8/site-packages/uriutils-0.1.16.dist-info/zip-safe
     955  Defl:N      520  46% 08-24-2021 15:53 f919d1a5  python/lib/python3.8/site-packages/uriutils-0.1.16.dist-info/RECORD
      92  Stored       92   0% 08-24-2021 15:53 aa32508b  python/lib/python3.8/site-packages/uriutils-0.1.16.dist-info/WHEEL
       9  Stored        9   0% 08-24-2021 15:53 18ff9e74  python/lib/python3.8/site-packages/uriutils-0.1.16.dist-info/top_level.txt
       0  Stored        0   0% 08-24-2021 15:53 00000000  python/lib/python3.8/site-packages/uriutils-0.1.16.dist-info/REQUESTED
       4  Stored        4   0% 08-24-2021 15:53 c2971fc7  python/lib/python3.8/site-packages/uriutils-0.1.16.dist-info/INSTALLER
    2284  Defl:N     1059  54% 08-24-2021 15:53 46dc4e0a  python/lib/python3.8/site-packages/uriutils-0.1.16.dist-info/METADATA

And relevant part from the script of the lambda function I'm trying to create:

bash-5.0# head -n 20 /tmp/localstack/lambda_script_l_ef732d3a.py
import logging
import os
import json
import time
import uuid
import io
from tempfile import NamedTemporaryFile
from uriutils import uri_read, uri_exists, uri_dump
from pdf2image import convert_from_path

from google.cloud import vision
from google.oauth2 import service_account

from vision_api import convert_to_analysis_response

import boto3
import base64
from botocore.exceptions import ClientError

Within the lambda container I can see the right contents in relevant directories:

bash-5.0# ls -la /tmp/localstack/lambda-with-layers.af8b42c5/
total 220
drwxr-xr-x    3 root     root          4096 Aug 25 15:08 .
drwxrwxrwx    1 localsta localsta      4096 Aug 25 15:08 ..
drwxr-xr-x    8 root     root          4096 Aug 25 15:08 __layers__
-rw-r--r--    1 root     root        151404 Aug 20 19:18 documents_pb2.py
-rwxr-xr-x    1 root     root           228 May 31 11:48 function.json
-rwxr-xr-x    1 root     root          5755 Aug 20 19:18 main.py
-rw-r--r--    1 root     root         20050 Aug 20 19:18 options_pb2.py
-rw-r--r--    1 root     root         19715 Aug 25 15:08 original_lambda_archive.zip
-rw-r--r--    1 root     root          2903 Jul 19 17:52 vision_api.py

bash-5.0# ls -la /tmp/localstack/lambda-with-layers.af8b42c5/__layers__/python/lib/python3.8/site-packages/ | grep uriutils
drwxr-xr-x    3 root     root          4096 Aug 24 18:53 uriutils
drwxr-xr-x    2 root     root          4096 Aug 24 18:53 uriutils-0.1.16.dist-info

Environment

- OS: MacOS
- LocalStack: 0.12.17

Anything else?

No response

Metadata

Metadata

Assignees

Labels

aws:lambdaAWS Lambdastatus: triage neededRequires evaluation by maintainerstype: bugBug report

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions