For RecipeYak’s existence, we’ve manually written the API queries in the UI. This gets tedious as there are lots of fields in both request params and responses. It’s also easy for the backend schema to get out of sync from the frontend, and it’s difficult to ensure API changes don’t break the UI’s requests.
Recently I decided it was time to simplify this setup and generate the API client.
There are two steps involved in the codegen process:
We use Pydantic to ensure static types are accurate at runtime. This means as long as we generate our schema off the types, we’ll never lie to the client.
Before codegen, endpoints didn’t have explicit types for params and responses:
@endpoint()
def ingredient_update_view(
request: AuthedHttpRequest, ingredient_id: int
) -> JsonResponse:
params = IngredientsPatchParams.parse_raw(request.body)
# --snip--
return JsonResponse(serialize_ingredient(ingredient))
To facilitate both static typing and runtime type introspection, I added generics to the request and response types:
@endpoint()
def ingredient_update_view(
request: AuthedHttpRequest[IngredientsPatchParams], ingredient_id: int
) -> JsonResponse[IngredientResponse]:
params = IngredientsPatchParams.parse_raw(request.body)
# --snip--
return JsonResponse(serialize_ingredient(ingredient))
It’s not perfect, since there’s nothing that enforces actually serializing the request params with the same type as the one specified in the request, but it’s a good v0.
Additionally, since we’re now statically typing the response shape, any non-successful response needs to be an exception.
I created a helper exception for this, so instead of having:
# --snip--
if some_condition:
return JsonResponse(
{"code": "invalid_params", "message": "url was not valid"},
status=400
)
return JsonResponse(success_data)
we use:
# --snip--
if some_condition:
raise APIError(code="invalid_params", message="url was not valid")
return JsonResponse(success_data)
Ideally the API would look more like:
@endpoint()
def ingredient_update_view(
request: AuthedHttpRequest, params: IngredientsPatchParams
) -> IngredientResponse:
# --snip--
return serialize_ingredient(ingredient)
URL path params would be included in the params
argument and the params
would automatically be deserialized using Pydantic’s parse_raw
method before
they’re passed to the endpoint function.
This also avoids JsonResponse
, since all API endpoints use JSON.
Supporting differing success status codes gets trickier without the
JsonResponse
wrapper, but we could either centrilaze on all endpoints returning
200 for success or maybe return a tuple of status code and data for non-200s.
Generating the schema is pretty straightfoward, we:
prettier
✨It ends up being around 400 lines of code, including comments, empty lines, etc.
The generated schema on the other hand currently tops out at almost 6k LOC, OpenAPI is not concise, but it’s easy to work with.
After generating the OpenAPI Schema, we generate our TypeScript API client.
The generated API client functions are straightforward:
// generated by recipeyak.api.base.codegen
import { http } from "@/apiClient"
export function ingredientUpdate(params: {
quantity?: string | null
name?: string | null
description?: string | null
position?: string | null
optional?: boolean | null
ingredient_id: number
}) {
return http<{
id: number
quantity: string
name: string
description: string
position: string
optional: boolean
}>({
url: "/api/v1/ingredients/{ingredient_id}/",
method: "patch",
params,
pathParamNames: ["ingredient_id"]
})
}
The http
function we’re using is a wrapper around axios
that handles templating the url path params, serializing dates, and omitting the path param from the request body.
For a given API method, we pass the OpenAPI schema to a recursive function, _json_schema_to_typescript_type
, that returns a string of TypeScript.
Some highlights from _json_schema_to_typescript_type
:
request types are treated differently from response types:
ReadonlyArray
instead of Array
for inputsundefined
datetime
params are typed as Date
, while date
s are typed as string
. JS not having different types for date
s vs datetime
s is annoying
we don’t convert from snake case to camel case, field names mirror the actual API
Python type | JS request type | JS response type |
---|---|---|
date |
string |
string |
datetime |
Date |
string |
T | U (union) |
T | U (union) |
T | U (union) |
bool |
boolean |
boolean |
None |
null |
null |
int |
number |
number |
str |
string |
string |
list[T] |
ReadonlyArray<T> |
Array<T> |
NotRequired / has default |
[field_name]?: T |
omitted |
dict[str, T] |
Record<string, T> |
Record<string, T> |
no response (204) | n/a | unknown |
There are a few new CI jobs that help keep everything in sync and nicely formatted:
check_missing_api_schema_changes
Ensure the OpenAPI schema doesn’t get out of sync with the API.
Schema generation has a --check
option that generates the schema and compares it against the existing schema to see if any changes are missing.
Ensure the TypeScript API client doesn’t get out of sync with the OpenAPI schema.
Codegen also has a --check
option that generates the TypeScript modules in a temp directory and compares the directory, and all its files, against the one in the repo; checking for missing, extranous, and out of date files.
Ensure the OpenAPI schema is following the spec via redocly lint
.
It wasn’t too hard to start generating the API client, especially when we’re only targeting TypeScript.
The end result in terms of LOC is manageable and only requires pydantic and the standard library.
Here are the PRs that encompass the codegen changes:
]]>Archive
or Delete
on a recipe doesn’t do anything, and you’d need to refresh before the buttons work.
I’ve noticed this happening for at least a year, probably longer, and couldn’t figure out why.
But I recently stumbled upon this Stack Overflow post:
Turns out, there’s a bug in Safari where history.pushState()
breaks alert()
, confirm()
, and prompt()
.
Recipe Yak uses history.pushState()
for navigation between pages.
Kind of annoying having to reimplement confirm()
, but better than having the buggy behavior.
@
/u/
/id/
or /profiles/
/user?id=
@
@
@
/profile
An even split – seems like newer sites prefer the prefix which saves a lot of headache.
]]>date
s and datetime
s, you only get:
So when serializing a datetime
to JSON, we have a choice to make.
We could use a unix timestamp, which fits in JSON’s Number type, but reading the raw timestamp is annoying, you have to plug it into another tool to make sense of it.
We could also serialize to a string using ISO 8601
So:
date(2024, 1, 18)
becomes:
"2024-01-18"
And
datetime(2024, 1, 18, 23, 22, 20, 873446, tzinfo=timezone.utc)
becomes
"2024-01-18T23:22:20.873446+00:00Z"
When parsing a date string in JS land you’ll want to use parseISO
from date-fns
instead of just passing it to a Date
constructor.
parseISO("2024-01-18")
gives:
Thu Jan 18 2024 00:00:00 GMT-0500 (Eastern Standard Time)
While:
new Date("2024-01-18")
gives:
Wed Jan 17 2024 19:00:00 GMT-0500 (Eastern Standard Time)
Which isn’t want you want!
The Date constructor in JS works out of the box for datetimes, just be mindful of time zones
In addition to being human readable, ISO 8601 strings are lexicographically sortable!
Use ISO 8601 for serializing dates, times, datetimes, durations (like Python’s timedelta
) in JSON.
https://nextjs.org/docs/app/building-your-application/upgrading/from-vite#migration-steps
Formatting is broken with the step numbers, lots of 1. 1. 1. 1.
Weird that the suggested file ending for the config file is .mjs
Time: Sun Dec 3, 12:20pm EST
❯ yarn add next@latest
yarn add v1.22.17
[1/4] 🔍 Resolving packages...
[2/4] 🚚 Fetching packages...
warning Pattern ["string-width@^4.1.0"] is trying to unpack in the same destination "/Users/steve/Library/Caches/Yarn/v6/npm-string-width-cjs-4.2.3-269c7117d27b05ad2e536830a8ec895ef9c6d010-integrity/node_modules/string-width-cjs" as pattern ["string-width-cjs@npm:string-width@^4.2.0"]. This could result in non-deterministic behavior, skipping.
warning Pattern ["wrap-ansi@^7.0.0"] is trying to unpack in the same destination "/Users/steve/Library/Caches/Yarn/v6/npm-wrap-ansi-cjs-7.0.0-67e145cff510a6a6984bdf1152911d69d2eb9e43-integrity/node_modules/wrap-ansi-cjs" as pattern ["wrap-ansi-cjs@npm:wrap-ansi@^7.0.0"]. This could result in non-deterministic behavior, skipping.
warning Pattern ["string-width@^4.2.3"] is trying to unpack in the same destination "/Users/steve/Library/Caches/Yarn/v6/npm-string-width-cjs-4.2.3-269c7117d27b05ad2e536830a8ec895ef9c6d010-integrity/node_modules/string-width-cjs" as pattern ["string-width-cjs@npm:string-width@^4.2.0"]. This could result in non-deterministic behavior, skipping.
warning Pattern ["string-width@^4.2.0"] is trying to unpack in the same destination "/Users/steve/Library/Caches/Yarn/v6/npm-string-width-cjs-4.2.3-269c7117d27b05ad2e536830a8ec895ef9c6d010-integrity/node_modules/string-width-cjs" as pattern ["string-width-cjs@npm:string-width@^4.2.0"]. This could result in non-deterministic behavior, skipping.
error next@14.0.3: The engine "node" is incompatible with this module. Expected version ">=18.17.0". Got "18.7.0"
error Found incompatible module.
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.
volta pin node@20
Favicon and related are configured differently than vite, with vite we use https://github.com/darkobits/vite-plugin-favicons which works well and is easy to configure!
Need to change the env var access & prefix, can’t configure it like we can with vite which is annoying.
We configure this in TypeScript and Vite, while Next.js will look at the tsconfig to set this up for us.
This doesn’t work with the "output": "export"
apparently which means we end up with preflight requests in dev – not great.
Vite supports setting up proxy rewrites
Unclear if we can use Vitest with Next.js.
next dev
❯ s/dev
- yarn next dev
yarn run v1.22.17
\$ /Users/steve/projects/recipeyak/frontend/node_modules/.bin/next dev
(node:26386) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `node --trace-warnings ...` to show where the warning was created)
/Users/steve/projects/recipeyak/frontend/next.config.js:21
export default nextConfig
^^^^^^
SyntaxError: Unexpected token 'export'
Add "type": "module"
to package.json
next dev
againGot a bunch of duplicate errors about postcss.config.js
(a file related to our tailwindcss setup) and also some warnings about the rewrites not being supported.
❯ s/dev
- yarn next dev
yarn run v1.22.17
\$ /Users/steve/projects/recipeyak/frontend/node_modules/.bin/next dev
⚠ Specified "rewrites" will not automatically work with "output: export". See more info here: https://nextjs.org/docs/messages/export-no-custom-routes
⚠ Specified "rewrites" will not automatically work with "output: export". See more info here: https://nextjs.org/docs/messages/export-no-custom-routes
▲ Next.js 14.0.3
- Local: http://localhost:3000
- Environments: .env
⚠ Specified "rewrites" will not automatically work with "output: export". See more info here: https://nextjs.org/docs/messages/export-no-custom-routes
⚠ Specified "rewrites" will not automatically work with "output: export". See more info here: https://nextjs.org/docs/messages/export-no-custom-routes
✓ Ready in 2.8s
○ Compiling /[[...slug]] ...
Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/steve/projects/recipeyak/frontend/postcss.config.js from /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/lib/find-config.js not supported.
postcss.config.js is treated as an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which declares all .js files in that package scope as ES modules.
Instead either rename postcss.config.js to end in .cjs, change the requiring code to use dynamic import() which is available in all CommonJS modules, or change "type": "module" to "type": "commonjs" in /Users/steve/projects/recipeyak/frontend/package.json to treat all .js files as CommonJS (using .mjs for all ES modules instead).
at mod.require (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/require-hook.js:64:28)
at findConfig (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/lib/find-config.js:58:20)
at async getPostCssPlugins (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/config/blocks/css/plugins.js:89:18)
at async /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/config/blocks/css/index.js:124:36
at async Object.resolveUrlLoader (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/loaders/resolve-url-loader/index.js:60:25) {
code: 'ERR_REQUIRE_ESM'
}
Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/steve/projects/recipeyak/frontend/postcss.config.js from /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/lib/find-config.js not supported.
postcss.config.js is treated as an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which declares all .js files in that package scope as ES modules.
Instead either rename postcss.config.js to end in .cjs, change the requiring code to use dynamic import() which is available in all CommonJS modules, or change "type": "module" to "type": "commonjs" in /Users/steve/projects/recipeyak/frontend/package.json to treat all .js files as CommonJS (using .mjs for all ES modules instead).
at mod.require (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/require-hook.js:64:28)
at findConfig (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/lib/find-config.js:58:20)
at async getPostCssPlugins (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/config/blocks/css/plugins.js:89:18)
at async /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/config/blocks/css/index.js:124:36
at async Object.resolveUrlLoader (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/loaders/resolve-url-loader/index.js:60:25) {
code: 'ERR_REQUIRE_ESM'
}
⨯ unhandledRejection: Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/steve/projects/recipeyak/frontend/postcss.config.js from /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/lib/find-config.js not supported.
postcss.config.js is treated as an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which declares all .js files in that package scope as ES modules.
Instead either rename postcss.config.js to end in .cjs, change the requiring code to use dynamic import() which is available in all CommonJS modules, or change "type": "module" to "type": "commonjs" in /Users/steve/projects/recipeyak/frontend/package.json to treat all .js files as CommonJS (using .mjs for all ES modules instead).
at mod.require (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/require-hook.js:64:28)
at findConfig (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/lib/find-config.js:58:20)
at async getPostCssPlugins (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/config/blocks/css/plugins.js:89:18)
at async /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/config/blocks/css/index.js:124:36
at async Object.resolveUrlLoader (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/loaders/resolve-url-loader/index.js:60:25) {
code: 'ERR_REQUIRE_ESM'
}
⨯ unhandledRejection: Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/steve/projects/recipeyak/frontend/postcss.config.js from /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/lib/find-config.js not supported.
postcss.config.js is treated as an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which declares all .js files in that package scope as ES modules.
Instead either rename postcss.config.js to end in .cjs, change the requiring code to use dynamic import() which is available in all CommonJS modules, or change "type": "module" to "type": "commonjs" in /Users/steve/projects/recipeyak/frontend/package.json to treat all .js files as CommonJS (using .mjs for all ES modules instead).
at mod.require (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/require-hook.js:64:28)
at findConfig (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/lib/find-config.js:58:20)
at async getPostCssPlugins (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/config/blocks/css/plugins.js:89:18)
at async /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/config/blocks/css/index.js:124:36
at async Object.resolveUrlLoader (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/loaders/resolve-url-loader/index.js:60:25) {
code: 'ERR_REQUIRE_ESM'
}
Try using export
instead of module.exports
:
export default {
plugins: {
tailwindcss: {},
autoprefixer: {}
}
}
But still erroring:
⨯ unhandledRejection: Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/steve/projects/recipeyak/frontend/postcss.config.js from /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/lib/find-config.js not supported.
Instead change the require of postcss.config.js in /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/lib/find-config.js to a dynamic import() which is available in all CommonJS modules.
at mod.require (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/require-hook.js:64:28)
at findConfig (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/lib/find-config.js:58:20)
at async getPostCssPlugins (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/config/blocks/css/plugins.js:89:18)
at async /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/config/blocks/css/index.js:124:36
at async Object.resolveUrlLoader (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/webpack/loaders/resolve-url-loader/index.js:60:25) {
code: 'ERR_REQUIRE_ESM'
}
Deleted postcss.config.js
and end up with:
Error: Page "/[[...slug]]/page" is missing exported function "generateStaticParams()", which is required with "output: export" config.
generateStaticParams()
But it complains about:
Page "/[[...slug]]/page" cannot use both "use client" and export function "generateStaticParams()".
❯ s/dev
- yarn next dev
yarn run v1.22.17
\$ /Users/steve/projects/recipeyak/frontend/node_modules/.bin/next dev
⚠ Specified "rewrites" will not automatically work with "output: export". See more info here: https://nextjs.org/docs/messages/export-no-custom-routes
⚠ Specified "rewrites" will not automatically work with "output: export". See more info here: https://nextjs.org/docs/messages/export-no-custom-routes
▲ Next.js 14.0.3
- Local: http://localhost:3000
- Environments: .env
⚠ Specified "rewrites" will not automatically work with "output: export". See more info here: https://nextjs.org/docs/messages/export-no-custom-routes
⚠ Specified "rewrites" will not automatically work with "output: export". See more info here: https://nextjs.org/docs/messages/export-no-custom-routes
✓ Ready in 2.1s
○ Compiling /[[...slug]] ...
⚠ ./node_modules/ably/node_modules/ws/lib/buffer-util.js
Module not found: Can't resolve 'bufferutil' in '/Users/steve/projects/recipeyak/frontend/node_modules/ably/node_modules/ws/lib'
Import trace for requested module:
./node_modules/ably/node_modules/ws/lib/buffer-util.js
./node_modules/ably/node_modules/ws/lib/receiver.js
./node_modules/ably/node_modules/ws/index.js
./node_modules/ably/build/ably-node.js
./node_modules/@ably-labs/react-hooks/dist/mjs/AblyReactHooks.js
./node_modules/@ably-labs/react-hooks/dist/mjs/index.js
./src/queries/cookChecklistFetch.ts
./src/pages/cook-detail/CookingFullscreen.tsx
./src/pages/cook-detail/CookDetail.page.tsx
./src/components/App.tsx
./src/app/[[...slug]]/page.tsx
./node_modules/ably/node_modules/ws/lib/validation.js
Module not found: Can't resolve 'utf-8-validate' in '/Users/steve/projects/recipeyak/frontend/node_modules/ably/node_modules/ws/lib'
Import trace for requested module:
./node_modules/ably/node_modules/ws/lib/validation.js
./node_modules/ably/node_modules/ws/lib/receiver.js
./node_modules/ably/node_modules/ws/index.js
./node_modules/ably/build/ably-node.js
./node_modules/@ably-labs/react-hooks/dist/mjs/AblyReactHooks.js
./node_modules/@ably-labs/react-hooks/dist/mjs/index.js
./src/queries/cookChecklistFetch.ts
./src/pages/cook-detail/CookingFullscreen.tsx
./src/pages/cook-detail/CookDetail.page.tsx
./src/components/App.tsx
./src/app/[[...slug]]/page.tsx
⨯ Error: Page "/[[...slug]]/page" is missing exported function "generateStaticParams()", which is required with "output: export" config.
at DevServer.renderToResponseWithComponentsImpl (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/base-server.js:1039:27)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async DevServer.renderPageComponent (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/base-server.js:1852:24)
at async DevServer.renderToResponseImpl (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/base-server.js:1890:32)
at async DevServer.pipeImpl (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/base-server.js:902:25)
at async NextNodeServer.handleCatchallRenderRequest (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/next-server.js:266:17)
at async DevServer.handleRequestImpl (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/base-server.js:798:17) {
page: '/'
}
⚠ ./node_modules/ably/node_modules/ws/lib/buffer-util.js
Module not found: Can't resolve 'bufferutil' in '/Users/steve/projects/recipeyak/frontend/node_modules/ably/node_modules/ws/lib'
Import trace for requested module:
./node_modules/ably/node_modules/ws/lib/buffer-util.js
./node_modules/ably/node_modules/ws/lib/receiver.js
./node_modules/ably/node_modules/ws/index.js
./node_modules/ably/build/ably-node.js
./node_modules/@ably-labs/react-hooks/dist/mjs/AblyReactHooks.js
./node_modules/@ably-labs/react-hooks/dist/mjs/index.js
./src/queries/cookChecklistFetch.ts
./src/pages/cook-detail/CookingFullscreen.tsx
./src/pages/cook-detail/CookDetail.page.tsx
./src/components/App.tsx
./src/app/[[...slug]]/page.tsx
./node_modules/ably/node_modules/ws/lib/validation.js
Module not found: Can't resolve 'utf-8-validate' in '/Users/steve/projects/recipeyak/frontend/node_modules/ably/node_modules/ws/lib'
Import trace for requested module:
./node_modules/ably/node_modules/ws/lib/validation.js
./node_modules/ably/node_modules/ws/lib/receiver.js
./node_modules/ably/node_modules/ws/index.js
./node_modules/ably/build/ably-node.js
./node_modules/@ably-labs/react-hooks/dist/mjs/AblyReactHooks.js
./node_modules/@ably-labs/react-hooks/dist/mjs/index.js
./src/queries/cookChecklistFetch.ts
./src/pages/cook-detail/CookingFullscreen.tsx
./src/pages/cook-detail/CookDetail.page.tsx
./src/components/App.tsx
./src/app/[[...slug]]/page.tsx
⚠ ./node_modules/ably/node_modules/ws/lib/buffer-util.js
Module not found: Can't resolve 'bufferutil' in '/Users/steve/projects/recipeyak/frontend/node_modules/ably/node_modules/ws/lib'
Import trace for requested module:
./node_modules/ably/node_modules/ws/lib/buffer-util.js
./node_modules/ably/node_modules/ws/lib/receiver.js
./node_modules/ably/node_modules/ws/index.js
./node_modules/ably/build/ably-node.js
./node_modules/@ably-labs/react-hooks/dist/mjs/AblyReactHooks.js
./node_modules/@ably-labs/react-hooks/dist/mjs/index.js
./src/queries/cookChecklistFetch.ts
./src/pages/cook-detail/CookingFullscreen.tsx
./src/pages/cook-detail/CookDetail.page.tsx
./src/components/App.tsx
./src/app/[[...slug]]/page.tsx
./node_modules/ably/node_modules/ws/lib/validation.js
Module not found: Can't resolve 'utf-8-validate' in '/Users/steve/projects/recipeyak/frontend/node_modules/ably/node_modules/ws/lib'
Import trace for requested module:
./node_modules/ably/node_modules/ws/lib/validation.js
./node_modules/ably/node_modules/ws/lib/receiver.js
./node_modules/ably/node_modules/ws/index.js
./node_modules/ably/build/ably-node.js
./node_modules/@ably-labs/react-hooks/dist/mjs/AblyReactHooks.js
./node_modules/@ably-labs/react-hooks/dist/mjs/index.js
./src/queries/cookChecklistFetch.ts
./src/pages/cook-detail/CookingFullscreen.tsx
./src/pages/cook-detail/CookDetail.page.tsx
./src/components/App.tsx
./src/app/[[...slug]]/page.tsx
⨯ Error: Page "/[[...slug]]/page" is missing exported function "generateStaticParams()", which is required with "output: export" config.
at DevServer.renderToResponseWithComponentsImpl (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/base-server.js:1039:27) {
page: '/favicon.ico'
}
⨯ Error: Page "/[[...slug]]/page" is missing exported function "generateStaticParams()", which is required with "output: export" config.
at DevServer.renderToResponseWithComponentsImpl (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/base-server.js:1039:27) {
page: '/apple-touch-icon-precomposed.png'
}
⨯ Error: Page "/[[...slug]]/page" is missing exported function "generateStaticParams()", which is required with "output: export" config.
at DevServer.renderToResponseWithComponentsImpl (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/base-server.js:1039:27) {
page: '/apple-touch-icon.png'
}
⨯ Error: Page "/[[...slug]]/page" is missing exported function "generateStaticParams()", which is required with "output: export" config.
at DevServer.renderToResponseWithComponentsImpl (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/base-server.js:1039:27) {
page: '/favicon.ico'
}
⚠ Fast Refresh had to perform a full reload due to a runtime error.
⨯ Error: Page "/[[...slug]]/page" cannot use both "use client" and export function "generateStaticParams()".
at getPageStaticInfo (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/analysis/get-page-static-info.js:460:19)
at async getStaticInfoIncludingLayouts (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/entries.js:105:28)
at async /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/dev/hot-reloader-webpack.js:642:50
at async Promise.all (index 0)
at async config.entry (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/dev/hot-reloader-webpack.js:607:17)
Error: Page "/[[...slug]]/page" cannot use both "use client" and export function "generateStaticParams()".
at getPageStaticInfo (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/analysis/get-page-static-info.js:460:19)
at async getStaticInfoIncludingLayouts (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/entries.js:105:28)
at async /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/dev/hot-reloader-webpack.js:642:50
at async Promise.all (index 0)
at async config.entry (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/dev/hot-reloader-webpack.js:607:17)
Error: Page "/[[...slug]]/page" cannot use both "use client" and export function "generateStaticParams()".
at getPageStaticInfo (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/analysis/get-page-static-info.js:460:19)
at async getStaticInfoIncludingLayouts (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/build/entries.js:105:28)
at async /Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/dev/hot-reloader-webpack.js:642:50
at async Promise.all (index 0)
at async config.entry (/Users/steve/projects/recipeyak/frontend/node_modules/next/dist/server/dev/hot-reloader-webpack.js:607:17)
generateStaticParams()
Ugh, the page doesn’t automatically refresh after updating the file!
Found a related error and a bug on the next.js issue tracker
Seems "next": "^14.0.3"
, has this bug, and we need to downgrade to next@13.4.19
.
But now we’re getting runtime errors about data being undefined – seems react-query isn’t working.
TypeError: undefined is not a function (near '...teams.data.map...')
[Error] The above error occurred in the <TeamSelect> component:
TeamSelect
div
styled.div
div
styled.div
UserDropdown
div
NavButtons
nav
styled.nav
Navbar
ContainerBase
NavPage
UserHome
HomePage
Component@
sentryRoute(Route)
Route
Component@
Component@
Component@
AppRouter
Component@
ErrorBoundary
DndProvider
Component@
Le
QueryClientProvider
PersistQueryClientProvider
App
Component@
profiler(App)
NoSSR
Suspense
LoadableComponent
Page
InnerLayoutRouter
Component@
RedirectBoundary
NotFoundBoundary
LoadingBoundary
ErrorBoundary
Component@
ScrollAndFocusHandler
RenderFromTemplateContext
OuterLayoutRouter
InnerLayoutRouter
Component@
RedirectBoundary
Component@
NotFoundBoundary
LoadingBoundary
ErrorBoundary
Component@
ScrollAndFocusHandler
RenderFromTemplateContext
OuterLayoutRouter
div
body
html
Component@
RedirectBoundary
Component@
NotFoundBoundary
DevRootNotFoundBoundary
PureComponent@
HotReload
Router
Component@
ErrorBoundary
AppRouter
ServerRoot
RSCComponent
Root
React will try to recreate this component tree from scratch using the error boundary you provided, ErrorBoundary.
BASE_URL
instead of rewritesUpdate the http client to have a BASE_URL
that points to localhost:8000
while next.js runs on localhost:3000
.
End up with a bunch of options requests which I was trying to avoid with the rewrites:
[03/Dec/2023 18:07:24] "OPTIONS /api/v1/recipes/recently_created HTTP/1.1" 403 58
Not Found: /api/v1/t/-1/calendar/
level=WARNING msg="Not Found: /api/v1/t/-1/calendar/" user_id=none request_id=671003da502d4e1e9eb150b22bd6676e name=django.request pathname="/Users/steve/projects/recipeyak/backend/.venv/lib/python3.11/site-packages/django/utils/log.py" lineno=224 funcname=log_response process=37918 thread=6161526784
[03/Dec/2023 18:07:24] "OPTIONS /api/v1/t/-1/calendar/?start=2023-12-03&end=2023-12-09 HTTP/1.1" 404 8530
Sun Dec 3, 1:10pm 🪦
Failed PR for postarity and a previous attempt back in Dec 2021.
Lots of friction trying to migrate from Vite to Next.js.
I think the docs for Vite are much better than Next.js and I find the entire experience of Vite to be much more polished.
Hopefully Vite will support SSR out of the box, or Next.js catches up with Vite in terms of polish and developer experience.
]]>yt-dlp -o "tiktok-videos/tiktok@%(uploader)s:%(id)s:%(title).100B.%(ext)s" https://www.tiktok.com/@june_banoon/video/6979637268126420230
which outputs:
[TikTok] Extracting URL: https://www.tiktok.com/@june_banoon/video/6979637268126420230
[TikTok] 6979637268126420230: Downloading video feed
[info] 6979637268126420230: Downloading 1 format(s): bytevc1_720p_1495094-2
[download] Destination: tiktok-videos/tiktok@june_banoon:6979637268126420230:He lives down the road and we call him Mashed Potatoes #fyp #TakisTransformation #cat #catsoftiktok.mp4
[download] 100% of 2.50MiB in 00:00:00 at 8.19MiB/s
However, if you want to download all of your favorited videos, you’ll have to do a lot of clicking around the UI to get all of the urls.
Turns out you can get a GDPR download of all your data which includes your favorited videos, liked videos, DMs, comments, etc. but it takes a few days to process so instead we can scrape the data!
(Also I wasn’t aware this was a thing until after I got everything working D;)
After Copying as cURL
and copious amounts of data munging, I arrived at tiktoker which can:
yt-dlp
It also saves the metadata to sqlite for further perusing and added robustness (an export can be paused part way through and resumed).
It works well so far, but there are a couple caveats:
And after getting it all working, I found there was existing prior art in this space:
]]>With proto3, every field is optional. This can be a problem if you want to ensure you always set certain fields.
For example, if we have an RPC to send a message defined as follows:
syntax = "proto3";
package com.my.pkg;
service MessageService {
rpc SendMessage(MessageRequest) returns (MessageResponse);
}
message MessageRequest {
string body = 1;
string from = 2;
string to = 3;
}
message MessageResponse {
string id = 1;
}
And we call it using Java:
import com.my.pkg.MessageRequest;
public class Main {
public static void main(String[] args) {
var client = new Client();
var msg = MessageRequest.newBuilder()
.setBody("hello world")
.setFrom("+15085550000")
.build();
var res = client.sendMessage(msg);
System.out.println("res", res);
}
}
We then compile our code, run our linters, which both pass without any problems.
Then when we run it, we’ll end up with an error because we forgot to set the to
field on the message!
Bummer.
One option is to write some code to validate the message before we send it over the wire.
import com.my.pkg.MessageRequest;
public class Main {
public static void main(String[] args) {
var client = new Client();
var msg = MessageRequest.newBuilder()
.setBody("hello world")
.setFrom("+15085550000")
.build();
for (var field : msg.getDescriptorForType().getFields()) {
if (!msg.hasField(field)) {
throw new RuntimeException("expected field to be set");
}
}
var res = client.sendMessage(msg);
System.out.println("res", res);
}
}
This sort of works, we could also even use Mockito and ArgumentCaptor to avoid running the check at runtime, but this still doesn’t handle cases like repeated
fields or nested messages.
Instead of inventing our own validation logic on top of protobufs, we could use the existing protovalidate package.
So instead our proto and code would look as follows:
syntax = "proto3";
package com.my.pkg;
service MessageService {
rpc SendMessage(MessageRequest) returns (MessageResponse);
}
message MessageRequest {
string body = 1 [(buf.validate.field).required = true];
string from = 2 [(buf.validate.field).string = {
min_len: 1
max_len: 15
}];
string to = 3 [(buf.validate.field).string = {
min_len: 1
max_len: 15
}];
}
message MessageResponse {
string id = 1;
}
package com.my.pkg;
import build.buf.protovalidate.results.ValidationException;
import com.my.pkg.MessageRequest;
public class Main {
public static void main(String[] args) {
var client = new Client();
var msg = MessageRequest.newBuilder()
.setBody("hello world")
.setFrom("+15085550000")
.build();
var validator = new Validator();
try {
var result = validator.validate(msg);
if (!result.violations.isEmpty()) {
System.out.println(result.toString());
System.exit(1);
}
} catch (ValidationException e) {
System.out.println("Validation failed: " + e.getMessage());
System.exit(1);
}
var res = client.sendMessage(msg);
System.out.println("res " + res.getId());
}
}
Check out protovalidate to help validate your protos!
]]>Some apps include unique ids in query params when sharing, which are pretty easy to remove as a user:
https://www.instagram.com/p/Cupv5rCM55G/?igshid=MTIzZWMxMTBkOA==
Stackoverflow
Some create a unique id and use that in the URL, which you can’t get around easily:
TikTok
Snapchat
Amazon
Some include urchin tags or similar but nothing unique:
https://twitter.com/Wikipedia/status/1690407300622696449?s=20
Some include nothing:
Youtube
So you’ll see things like:
String content = "some string content";
Which is surprising coming from more dynamic lands (Python, Ruby, TypeScript) or more modern static languages (Rust).
As of Java 10, the language has var
which allows for omitting the explicit type.
There’s even a rule in errorprone to prefer using var
in some cases.
But ultimately I don’t think errorprone rule goes far enough, it warns about boilerplate usages like:
CustomerCreateParams params =
CustomerCreateParams
.builder()
.setDescription("Example description")
.setEmail("test@example.com")
.setPaymentMethod("pm_card_visa")
.build();
but doesn’t warn about:
String foo = "foo";
Instead, I think var
should be preferred whenever possible, providing Java with readability similar to TypeScript.
So instead of:
String foo = "foo";
List<Character> charList = new ArrayList<>();
for (char c : foo.toCharArray()) {
System.out.println(c);
charList.add(c);
}
we’d have:
var foo = "foo";
var charList = new ArrayList<Character>();
for (var c : foo.toCharArray()) {
System.out.println(c);
charList.add(c);
}
System.out.println(charList.size());
Additionally, in some cases by using var
you can avoid having to import the explicit types!
Java has another keyword called final
which makes variables immutable,
similar to Javascript’s const
.
So the initial code example would look more like:
final String content = "some string content";
Even more verbose!
Instead of using final
everywhere, and leaving it off when we want to have a mutable variable, we can use Errorprone’s Var rule which eliminates most usages of final
and assumes all variables are final
, unless annotated with @Var
.
So the code sample would be:
var content = "some string content";
or, if we wanted to mutate the content:
@Var var content = "some string content";
if (someCondition) {
content = "other content";
}
Use var
and Errorprone’s Var rule for more concise Java.
By default, each Celery worker prefetches 4 jobs from the queue.
This means you could have one job that takes 45 minutes block 3 other jobs that complete in under a second.
If a task raises an exception, or a worker process dies, Celery will by default lose the job.
So if you happen to reboot or redeploy, any running jobs with be lost to the sands of time.
task_acks_late
and task_reject_on_worker_lost
Celery doesn’t default to using exponential backoff for job retries.
Task.retry_backoff
Sometimes called transactionally staged jobs, or the transactional outbox pattern, but the gist is you can’t enqueue a job inside a transaction.
This means you can’t have an HTTP request handler that creates a new User
and schedules a job to send them an email, because the database save might succeed but the job might not persist, or vice versa.
Since Celery doesn’t have transactional job enqueuing, canvas, chords and friends are a recipe for losing jobs or having broken workflows.
For example:
@shared_task(name="tasks.add")
def add(a: int, b: int) -> int:
return a + b
add.signature(args=(1, 2)).delay()
add.s(1, 2).apply_async(countdown=60, expires=120)
signature('tasks.add', args=(1, 2)).delay()
Another gripe is bind=True
.
If you want to access Celery’s builtin retry methods (along with additional context) you need to use the bind
kwarg
in your task definition, which causes Celery to pass in a first argument automatically.
This behavior is tricky to type check and results in an inconsistent API.
When you want to configure some cron jobs the config isn’t type safe:
app.conf.beat_schedule = {
'add-every-30-seconds': {
'task': 'tasks.add',
'schedule': 30.0,
'args': (16, 16)
}
}
app.conf.timezone = 'UTC'
For example, you write your function and ship it to production.
@shared_task(name="tasks.send_email")
def send_email(to: str) -> None:
...
It’s running for a while and then you realize you want to support CC
ing, so you update your function signature:
@shared_task(name="tasks.send_email")
def send_email(to: list[str], cc: list[str]) -> None:
...
and you deploy to production and immediately start getting runtime exceptions.
The problem is you have existing workers running with the old task version, and existing jobs serialized in the queue, which are incompatible with the new task version, so you end up with a bunch of exceptions.
Celery doesn’t help you enforce safe evolution of tasks. The correct way to update the task would be to create a new version or add some optional params.
Also, Celery doesn’t enforce defining a name
for each task so if you let Celery auto-generate it, and then move a task definition to a different file, you’ll end up with runtime exceptions.
name
to your task definitions.If your jobs aren’t interruptible and you have a job that takes 45 minutes to complete, then you have to wait 45 minutes before deploying a new version.
Celery doesn’t have any builtin functionality to support this so you have to roll your own.
Whether it’s a cron job or a manually scheduled job, there isn’t a builtin way to disable a job.
Celery doesn’t support asyncio, so you’re out of luck if you’re using async
.
async
calls with asyncio.run()
.When looking into testing for Celery you’ll see the task_always_eager
option pop up, which shouldn’t be used since it skips the Celery specific serialization and job storage.
The docs also suggest relying on mocking which results in brittle tests.
It’s easy to write a Celery task that takes an argument that isn’t serializable and you won’t know until runtime.
Another gotcha is using pickle
instead of json
for serialization. pickle
can serialize more objects by default but comes with its own caveats.
There are docs for monitoring but Celery is pretty limited in what it provides by default.
The Celery internals aren’t type checked and there aren’t types for public APIs. You can use celery-types, but it’s limited in strictness.
If you’re project is heavily invested in Celery, it’s tricky to migrate, so implementing the available fixes might be best.
For a new project, I wouldn’t use Celery. The biggest issue is how easy it is to lose jobs. Instead I’d use a transactional outbox setup with some workers.
Also worth looking into the new hotness, Temporal, but it has its own learning curve.
In general, be careful how you enqueue your jobs and ensure changes to your tasks are backwards compatible.
]]>