Failure to Build Features from BigQuery

I set up a project with a BigQuery data source. However, it fails to build features, which happens after a long time trying to do so. I have attached a screenshot of the build process.

I am running version 0.6.9 of Layer.

On my Layer dashboard, I am able to see the features SQL.

Screenshot from 2021-08-02 10-41-51

Screenshot from 2021-08-02 10-42-22

The git repo

Hey, Eric.

Thanks for reporting. I’ve checked the data source you’ve created for the big query and it looks like it is named layer-bigquery. However, in dataset/dataset.yml and features/dataset.yml you’re using layer. Changing it to layer-bigquery should fix the issue.

On the other hand, such error should be reported and there’s a bug on our side. We plan to release a fix this week.

Thank you.

I renamed the target, but now I run into an error without any clue as to what could be causing it. There is no indication of what is throwing the LayerClientException…

Hi @gitonga
We are still looking into it.
In the meantime, can you try the following:
First, layer 0.7.0 is now released. Please upgrade it;
After that, try to re-run your project and in case it fails attach the full console output(as before but also including run id that is printed in the beginning).

Lastly, can you run(either before or after the run) layer diagnose and add it’s output? It will help us look through backend logs because we will understand your timezone, etc.

Hi. I did try with the latest version, and still failed. Let me re-run it and attach the information you request.

Run session id: 3d9b8567-5e23-4604-a5e1-56c413553db8

Local diagnostics for Layer 0.7.0
OS: Linux 5.11.0-25-generic
Platform: uname_result(system='Linux', node='delton', release='5.11.0-25-generic', version='#27~20.04.1-Ubuntu SMP Tue Jul 13 17:41:23 UTC 2021', machine='x86_64', processor='x86_64')
Interpreter path: /home/gitonga/Develop/Data Science/Projects/Rabbit Holes/Writing/Mine/clients/layer/bin/python3
Interpreter version: 3.8.10 (default, Jun  2 2021, 10:49:15) 
[GCC 9.4.0]
Timezone: EAT
Layer project found in the current directory. Printing the directory structure:
β”œβ”€β”€ models
β”‚   └── model
β”‚       β”œβ”€β”€ requirements.txt
β”‚       β”œβ”€β”€ model.yml
β”‚       └──
β”œβ”€β”€ ld
β”œβ”€β”€ .layer
β”‚   └── project.yml
β”œβ”€β”€ data
β”‚   β”œβ”€β”€ features
β”‚   β”‚   β”œβ”€β”€ success.sql
β”‚   β”‚   β”œβ”€β”€ month.sql
β”‚   β”‚   β”œβ”€β”€ day.sql
β”‚   β”‚   β”œβ”€β”€ rated.sql
β”‚   β”‚   β”œβ”€β”€ dataset.yml
β”‚   β”‚   β”œβ”€β”€ number_of_directors.sql
β”‚   β”‚   β”œβ”€β”€ release_date.sql
β”‚   β”‚   └── runtime.sql
β”‚   └── dataset
β”‚       └── dataset.yaml
β”œβ”€β”€ .git
β”‚   β”œβ”€β”€ COMMIT_EDITMSG
β”‚   β”œβ”€β”€ hooks
β”‚   β”‚   β”œβ”€β”€ post-update.sample
β”‚   β”‚   β”œβ”€β”€ pre-receive.sample
β”‚   β”‚   β”œβ”€β”€ prepare-commit-msg.sample
β”‚   β”‚   β”œβ”€β”€ fsmonitor-watchman.sample
β”‚   β”‚   β”œβ”€β”€ pre-merge-commit.sample
β”‚   β”‚   β”œβ”€β”€ pre-commit.sample
β”‚   β”‚   β”œβ”€β”€ update.sample
β”‚   β”‚   β”œβ”€β”€ pre-rebase.sample
β”‚   β”‚   β”œβ”€β”€ pre-push.sample
β”‚   β”‚   β”œβ”€β”€ commit-msg.sample
β”‚   β”‚   β”œβ”€β”€ applypatch-msg.sample
β”‚   β”‚   └── pre-applypatch.sample
β”‚   β”œβ”€β”€ description
β”‚   β”œβ”€β”€ info
β”‚   β”‚   └── exclude
β”‚   β”œβ”€β”€ HEAD
β”‚   β”œβ”€β”€ logs
β”‚   β”‚   β”œβ”€β”€ HEAD
β”‚   β”‚   └── refs
β”‚   β”‚       β”œβ”€β”€ heads
β”‚   β”‚       β”‚   └── main
β”‚   β”‚       └── remotes
β”‚   β”‚           └── origin
β”‚   β”‚               └── main
β”‚   β”œβ”€β”€ branches
β”‚   β”œβ”€β”€ index
β”‚   β”œβ”€β”€ refs
β”‚   β”‚   β”œβ”€β”€ heads
β”‚   β”‚   β”‚   └── main
β”‚   β”‚   β”œβ”€β”€ remotes
β”‚   β”‚   β”‚   └── origin
β”‚   β”‚   β”‚       └── main
β”‚   β”‚   └── tags
β”‚   β”œβ”€β”€ config
β”‚   └── objects
β”‚       β”œβ”€β”€ 53
β”‚       β”‚   └── cfac1ef716416209f50a93c3a63fb1a2ae83b7
β”‚       β”œβ”€β”€ bb
β”‚       β”‚   β”œβ”€β”€ e8ee49901432b79b4a49d9731b22abb9ab96a5
β”‚       β”‚   └── 92ce7994bf8c97b4da0255a8ce073306846151
β”‚       β”œβ”€β”€ info
β”‚       β”œβ”€β”€ 1f
β”‚       β”‚   └── 2961e802b0539716a6af24079c8d5f75cc305a
β”‚       β”œβ”€β”€ 98
β”‚       β”‚   └── b1ad4f88eaed1203d778c1cac1aeb5c1c1b0e6
β”‚       β”œβ”€β”€ ef
β”‚       β”‚   └── f2d5263410988aec4371ff8e8e63f7533efa9e
β”‚       β”œβ”€β”€ f7
β”‚       β”‚   β”œβ”€β”€ 2660e2e479745aa1a1be8580ff10055d1f1a2b
β”‚       β”‚   └── 31b0b8061cdee80500ed1652d23f4013a43bb3
β”‚       β”œβ”€β”€ 9b
β”‚       β”‚   └── 90dc5340f9b5f18c9208e3e98420a51e636c01
β”‚       β”œβ”€β”€ 4a
β”‚       β”‚   └── 74672c3a3aa9680c8d0a6f1dabb0fdd40daad4
β”‚       β”œβ”€β”€ pack
β”‚       β”œβ”€β”€ 32
β”‚       β”‚   └── 0769546e7e7881ae6d29f7eb7c9c9e204164de
β”‚       β”œβ”€β”€ c2
β”‚       β”‚   └── 9365031579da64a2e984b78ffb10db84190d74
β”‚       β”œβ”€β”€ e7
β”‚       β”‚   └── cc5312927cdc5d73efecb3ba677ccd781fee06
β”‚       β”œβ”€β”€ d5
β”‚       β”‚   └── 5304d832f680e0cc833620e839315da52e3d0f
β”‚       β”œβ”€β”€ c9
β”‚       β”‚   └── 6918a753845b117876df729eb62b458e735660
β”‚       β”œβ”€β”€ 8a
β”‚       β”‚   β”œβ”€β”€ 0a7fe3a4e8919128bdb8c96218697f6a05f5cd
β”‚       β”‚   └── a18a00b6a543a55cc876c0196e9eaa658d3b20
β”‚       β”œβ”€β”€ 4b
β”‚       β”‚   β”œβ”€β”€ 825dc642cb6eb9a060e54bf8d69288fbee4904
β”‚       β”‚   └── c45fece64fa08c8fc68d0edbe658ebb7889a39
β”‚       β”œβ”€β”€ 69
β”‚       β”‚   └── cd92992aa63b2d3d75ace2ffd722553845abfe
β”‚       β”œβ”€β”€ e3
β”‚       β”‚   └── ac850747d0d7ed8b7fbf7d93c428164d35dad7
β”‚       β”œβ”€β”€ ec
β”‚       β”‚   └── 72c259c9e1520c9e897e6bdff0ef9fabb7ac13
β”‚       β”œβ”€β”€ 37
β”‚       β”‚   └── cd344d0f8d710768cb9e229d37c2806a1ea717
β”‚       β”œβ”€β”€ 1c
β”‚       β”‚   └── b3fa37dc92392a9d532f8a9d740200d64ea749
β”‚       β”œβ”€β”€ 9c
β”‚       β”‚   └── 3641b49076838ff7c648e6c8844813d2c9b611
β”‚       β”œβ”€β”€ e5
β”‚       β”‚   └── f3cfe55e31e1b1643ba5d9626f7b8eea8b4f20
β”‚       β”œβ”€β”€ 79
β”‚       β”‚   └── 142d54829a7e42c213ed4e037116675ba7145d
β”‚       β”œβ”€β”€ 08
β”‚       β”‚   └── 74e1576524342af4d03052be11a406d52123ec
β”‚       β”œβ”€β”€ c8
β”‚       β”‚   └── e611f2c333dbb3aae916cdb6d32946cc4860ae
β”‚       β”œβ”€β”€ 55
β”‚       β”‚   └── a34adc4fbee09bb4a7fc0e88ea69749b330090
β”‚       β”œβ”€β”€ 27
β”‚       β”‚   └── cc7988877a577ef8cf05ce3b3f657ffaef4b1c
β”‚       β”œβ”€β”€ 5d
β”‚       β”‚   └── 2701487f4106704acc04a07e14774e7017ce2d
β”‚       β”œβ”€β”€ 8e
β”‚       β”‚   └── a945cbd4af2e4a7f15bfe3302b13e20290c163
β”‚       β”œβ”€β”€ 1e
β”‚       β”‚   └── d0e42fa866dc68f4a86290c35561ec013a20c8
β”‚       β”œβ”€β”€ 7a
β”‚       β”‚   └── 9645b59e3096eaf36e27231e6e33948b4bf55d
β”‚       β”œβ”€β”€ 80
β”‚       β”‚   β”œβ”€β”€ 45106745ce7f244c8c98711fa63b503682ef33
β”‚       β”‚   └── a965239cffe5237323c676b254730666b96536
β”‚       β”œβ”€β”€ 3d
β”‚       β”‚   └── 74b660f0484c10b3508bd1c9ffcd796731fdfc
β”‚       β”œβ”€β”€ 1a
β”‚       β”‚   └── a8c41512b1c27e8605f867c3e9dd13aa6aeb58
β”‚       β”œβ”€β”€ b9
β”‚       β”‚   └── 733ef5423aaba3a9833ef266430acf89d99888
β”‚       β”œβ”€β”€ 92
β”‚       β”‚   β”œβ”€β”€ b5d612f0fa55ea818115f2888f2100af7505d6
β”‚       β”‚   └── 5a8397193fe649e18149cba0a2b8c948add59c
β”‚       β”œβ”€β”€ ab
β”‚       β”‚   β”œβ”€β”€ 3bc43670b49a65d315694678bad01e1a87ed69
β”‚       β”‚   └── 446a9de943c5c2a5c3ef8d30b963c5e89be966
β”‚       β”œβ”€β”€ 26
β”‚       β”‚   └── 41651495691377ce57a738fb148143fa453331
β”‚       β”œβ”€β”€ 5b
β”‚       β”‚   └── c5e1a3aaca1cca176e6607b2a5365501aa898f
β”‚       β”œβ”€β”€ 76
β”‚       β”‚   β”œβ”€β”€ 7be6d416e55f2cb098890c6ea1f4018640b310
β”‚       β”‚   └── 1bb3a90f210f5664dc0dc635cf546a8bbb2f6b
β”‚       β”œβ”€β”€ f4
β”‚       β”‚   └── 3905c2c388317b4a49e79b5f071d893dd063c4
β”‚       β”œβ”€β”€ 7e
β”‚       β”‚   └── bfb267a17c5c8b410fb2c5949298bbfb9000ba
β”‚       β”œβ”€β”€ a7
β”‚       β”‚   └── a7ac0b2acc5cc4fe857f69661b7d4b239a9a1c
β”‚       └── 13
β”‚           └── 473094fdb4365364a01135895168e75ed2807c
/home/gitonga/.layer is found. Contents:
└── config.json
Installed packages: Babel 2.9.1 Flask 2.0.1 GitPython 3.1.14 Jinja2 3.0.1 Mako 1.1.4 MarkupSafe 2.0.1 Pillow 8.3.1 PyJWT 1.7.1 PyQt5 5.12.3 PyQt5-sip 12.9.0 PyQtWebEngine 5.12.1 PyYAML 5.4.1 Pygments 2.9.0 QDarkStyle 3.0.2 QtAwesome 1.0.3 QtPy 1.9.0 Rtree 0.9.7 SQLAlchemy 1.4.20 SecretStorage 3.3.1 Send2Trash 1.7.1 Sphinx 4.0.3 Werkzeug 2.0.1 aiodns 3.0.0 aiodocker 0.19.1 aiohttp 3.7.4.post0 alabaster 0.7.12 alembic 1.4.1 altair 4.1.0 appdirs 1.4.4 argon2-cffi 20.1.0 arrow 1.1.1 astor 0.8.1 astroid 2.6.2 async-generator 1.10 async-timeout 3.0.1 atomicwrites 1.4.0 attrs 21.2.0 autopep8 1.5.5 backcall 0.2.0 base58 2.1.0 binaryornot 0.4.4 black 21.6b0 bleach 3.3.0 blinker 1.4 boto3 1.17.106 botocore 1.20.106 brotlipy 0.7.0 cachetools 4.2.2 cchardet 2.1.7 certifi 2021.5.30 cffi 1.14.5 chardet 4.0.0 click 8.0.1 cloudpickle 1.6.0 colorama 0.4.4 commonmark 0.9.1 cookiecutter 1.7.3 cryptography 3.4.7 databricks-cli 0.14.3 debugpy 1.3.0 decorator 4.4.2 defusedxml 0.7.1 diff-match-patch 20200713 docker 5.0.0 docutils 0.17.1 entrypoints 0.3 filelock 3.0.12 flake8 3.8.4 gitdb 4.0.7 greenlet 1.1.0 grpcio 1.39.0 grpcio-tools 1.39.0 gunicorn 20.1.0 huggingface-hub 0.0.12 idna 2.10 imagesize 1.2.0 importlib-metadata 4.6.1 inflection 0.5.1 intervaltree 3.1.0 ipykernel 6.0.1 ipython 7.25.0 ipython-genutils 0.2.0 ipywidgets 7.6.3 isort 5.9.1 itsdangerous 2.0.1 jedi 0.17.2 jeepney 0.6.0 jinja2-time 0.2.0 jmespath 0.10.0 joblib 1.0.1 jsonschema 3.2.0 jupyter-client 6.1.12 jupyter-core 4.7.1 jupyterlab-pygments 0.1.2 jupyterlab-widgets 1.0.0 keyring 23.0.1 layer-sdk 0.7.0 lazy-object-proxy 1.6.0 matplotlib-inline 0.1.2 mccabe 0.6.1 mistune 0.8.4 mlflow 1.18.0 multidict 5.1.0 mypy-extensions 0.4.3 nbclient 0.5.3 nbconvert 6.1.0 nbformat 5.1.3 nest-asyncio 1.5.1 networkx 2.5.1 notebook 6.4.0 numpy 1.21.0 numpydoc 1.1.0 packaging 21.0 pandas 1.3.0 pandocfilters 1.4.3 parso 0.7.0 pathspec 0.8.1 pexpect 4.8.0 pickleshare 0.7.5 pip 21.2.2 pkg-resources 0.0.0 pluggy 0.13.1 polling 0.3.2 poyo 0.5.0 prometheus-client 0.11.0 prometheus-flask-exporter 0.18.2 prompt-toolkit 3.0.19 protobuf 3.17.3 psutil 5.8.0 ptyprocess 0.7.0 py4j 0.10.9 pyarrow 4.0.1 pycares 4.0.0 pycodestyle 2.6.0 pycparser 2.20 pydeck 0.6.2 pydocstyle 6.1.1 pyflakes 2.2.0 pylint 2.9.3 pyls-black 0.4.7 pyls-spyder 0.3.2 pyparsing 2.4.7 pyrsistent 0.18.0 pyspark 3.1.2 python-dateutil 2.8.1 python-editor 1.0.4 python-jsonrpc-server 0.4.0 python-language-server 0.36.2 python-slugify 5.0.2 pytz 2021.1 pyxdg 0.27 pyzmq 22.1.0 qstylizer 0.2.0 qtconsole 5.1.1 querystring-parser 1.2.4 regex 2021.7.6 requests 2.25.1 rich 10.5.0 rope 0.19.0 s3transfer 0.4.2 sacremoses 0.0.45 scikit-learn 0.24.2 scipy 1.7.0 setuptools 44.0.0 six 1.16.0 sklearn 0.0 smmap 4.0.0 snowballstemmer 2.1.0 sortedcontainers 2.4.0 sphinxcontrib-applehelp 1.0.2 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 spyder 5.0.5 spyder-kernels 2.0.5 sqlparse 0.4.1 tabulate 0.8.9 terminado 0.10.1 testpath 0.5.0 text-unidecode 1.3 textdistance 4.2.1 threadpoolctl 2.1.0 three-merge 0.1.1 tinycss2 1.1.0 tokenizers 0.10.3 toml 0.10.2 toolz 0.11.1 tornado 6.1 tqdm 4.61.2 traitlets 5.0.5 transformers 4.8.2 typing-extensions tzlocal 2.1 ujson 4.0.2 urllib3 1.26.6 validate-email 1.3 validators 0.18.2 watchdog 2.1.3 watermark 2.2.0 wcwidth 0.2.5 webencodings 0.5.1 websocket-client 1.1.0 wheel 0.34.2 widgetsnbextension 3.5.1 wrapt 1.12.1 wurlitzer 2.1.0 xgboost 1.4.2 yapf 0.31.0 yarl 1.6.3 zipp 3.5.0

Hi @gitonga

Apologies for delayed response

Can you please update materialization section of bq-movies/dataset.yml at main Β· ericgitonga/bq-movies Β· GitHub similar to bq-movies/dataset.yaml at main Β· ericgitonga/bq-movies Β· GitHub

Please let us know if it still fails.

I just tried that, and it still fails…

Any update on this issue?

Hi Eric,

Apologies for the delay here. We are trying to reproduce the issue.
Let me get back to you with an update.

kind regards, Gerard.

Hi Gerad,

Okay. Thank you.

Hi Eric,

Would you have a data sample or a link to the dataset you are using? I’d like to upload it to our BQ instance to reproduce this issue.

Thanks, Gerard.

Here it is:

Hi Eric,
Thanks for the data sample. Let me add it to our BQ instance. I’ll get back to you with any findings.

kr, Gerard.

Hi Eric,

A quick reply to let you know that we have found related issues with BigQuery and we are working on a fix for it.
I’ll keep you updated.

kind regards, Gerard.

1 Like

Thank you for the update Gerard.

Hey @gitonga, we’re still working on this bugfix, and will keep you updated as we make progress. Thanks!

Thank you for the update @volkan