Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhanced typings with optional environment variables #194

Merged
merged 14 commits into from Nov 28, 2022
Merged

Enhanced typings with optional environment variables #194

merged 14 commits into from Nov 28, 2022

Conversation

jsamr
Copy link
Contributor

@jsamr jsamr commented Nov 13, 2022 •
edited

This PR offers three major improvements:

Remark Should probably considered breaking for TS users and shipped within a major release.

Changes

Supports optional presence of environment variables

const cleaned = cleanEnv(env, {
  STR: str(),
  STR_OPT: str({ default: undefined }),
  STR_CHOICES: str({ choices: ["foo", "bar"] }),
  STR_REQ: str({ default: "foo" }),
  STR_DEFAULT_CHOICES: str({ default: "foo", choices: ["foo", "bar"] }),
});

Output type (VScode)
image

Notice a few things:

  • STR_OPT property is of type string | undefined
  • STR_CHOICES is narrowed down to 'foo' | 'bar'
  • STR_REQ is widened to type string while STR_DEFAULT_CHOICES is narrowed-down to 'foo' | 'bar'

Better JSON support

JSON output type can now be inferred from usage (default):

const cleaned = cleanEnv(env, {
  JSON_ANY: json(),
  JSON_REQ_ANY: json({ default: {} as any }),
  JSON_DEFAULT: json({ default: { foo: 'bar' } }),
  JSON_DEV_DEFAULT: json({ devDefault: { foo: 'bar' } }),
  JSON_DEFAULT_OPT: json<{ foo: 'bar' }>({ default: undefined }),
})

Output type (VScode)
image

Make Validator Wrappers

There are now three types of validators which can be created with distinct functions. These functions are identical runtime-wise, but provide fine-grained typings to cover different scenarios:

  • makeBaseValidator<BaseT> when you want the output to be narrowed-down to a subtype of BaseT (this is used in str and all string-based internal validators).
  • makeExactValidator<T> when you want the output to be widened to T (this is used in bool validator).
  • makeMarkupValidator for input which can produce arbitrary output types (this is used in json validator).

makeValidator is still there, but is typed like makeBaseValidator.

Guarantee types stability with thorough types testing

Thanks to expect-type, we can test all scenarios and maintain high types quality.

envalid/tests/types.test.ts

Lines 1 to 299 in a0f5e31

import {
cleanEnv,
str,
bool,
num,
RequiredValidatorSpec,
OptionalValidatorSpec,
makeMarkupValidator,
json,
makeBaseValidator,
} from '../src'
import { expectTypeOf } from 'expect-type'
describe('validators types', () => {
test('boolean validator', () => {
const validator = bool
expectTypeOf(validator()).toEqualTypeOf<RequiredValidatorSpec<boolean>>()
expectTypeOf(
validator({
default: false,
}),
).toEqualTypeOf<RequiredValidatorSpec<boolean>>()
expectTypeOf(
validator({
choices: [true, false],
default: true,
}),
).toEqualTypeOf<RequiredValidatorSpec<boolean>>()
expectTypeOf(validator({ default: undefined })).toEqualTypeOf<OptionalValidatorSpec<boolean>>()
expectTypeOf(validator({ devDefault: undefined })).toEqualTypeOf<
RequiredValidatorSpec<boolean>
>()
expectTypeOf(validator({ devDefault: false })).toEqualTypeOf<RequiredValidatorSpec<boolean>>()
})
test('number-based validators', () => {
const validator = makeBaseValidator<number>(() => 1)
// Specifying default or devDefault value should cause validator spec type param to widen
expectTypeOf(
validator({
default: 0,
}),
).toEqualTypeOf<RequiredValidatorSpec<number>>()
expectTypeOf(
validator({
devDefault: 0,
}),
).toEqualTypeOf<RequiredValidatorSpec<number>>()
// But this inference can be overridden by specifying a type parameter
expectTypeOf(
validator<0>({
default: 0,
}),
).toEqualTypeOf<RequiredValidatorSpec<0>>()
expectTypeOf(
validator<0>({
devDefault: 0,
}),
).toEqualTypeOf<RequiredValidatorSpec<0>>()
// Choices
expectTypeOf(
validator({
choices: [1, 2],
}),
).toEqualTypeOf<RequiredValidatorSpec<1 | 2>>()
expectTypeOf(
validator({
choices: [1, 2],
default: 1,
}),
).toEqualTypeOf<RequiredValidatorSpec<1 | 2>>()
// @ts-expect-error - 3 is not assignable to 1 | 2
validator({ choices: [1, 2], default: 3 })
// @ts-expect-error - 3 is not assignable to 1 | 2
validator({ choices: [1, 2], devDefault: 3 })
// Basic
expectTypeOf(validator()).toEqualTypeOf<RequiredValidatorSpec<number>>()
expectTypeOf(validator<1>()).toEqualTypeOf<RequiredValidatorSpec<1>>()
expectTypeOf(validator({ default: undefined })).toEqualTypeOf<OptionalValidatorSpec<number>>()
expectTypeOf(validator({ devDefault: undefined })).toEqualTypeOf<
RequiredValidatorSpec<number>
>()
expectTypeOf(validator<2>({ devDefault: 2 })).toEqualTypeOf<RequiredValidatorSpec<2>>()
})
test('string-based validators', () => {
const validator = makeBaseValidator<string>(() => '')
// Specifying default or devDefault value should cause validator spec type param to widen
expectTypeOf(
validator({
default: 'foo',
}),
).toEqualTypeOf<RequiredValidatorSpec<string>>()
expectTypeOf(
validator({
devDefault: 'foo',
}),
).toEqualTypeOf<RequiredValidatorSpec<string>>()
// But this inference can be overridden by specifying a type parameter
expectTypeOf(
validator<'foo'>({
default: 'foo',
}),
).toEqualTypeOf<RequiredValidatorSpec<'foo'>>()
expectTypeOf(
validator<'foo'>({
devDefault: 'foo',
}),
).toEqualTypeOf<RequiredValidatorSpec<'foo'>>()
expectTypeOf(
validator({
choices: ['foo', 'bar'],
}),
).toEqualTypeOf<RequiredValidatorSpec<'foo' | 'bar'>>()
expectTypeOf(
validator({
choices: ['foo', 'bar'],
default: 'foo',
}),
).toEqualTypeOf<RequiredValidatorSpec<'foo' | 'bar'>>()
//@ts-expect-error - baz is not assignable to 'foo' | 'bar'
validator({ choices: ['foo', 'bar'], default: 'baz' })
// Basic
expectTypeOf(validator()).toEqualTypeOf<RequiredValidatorSpec<string>>()
expectTypeOf(validator<'foo'>()).toEqualTypeOf<RequiredValidatorSpec<'foo'>>()
expectTypeOf(validator({ default: undefined })).toEqualTypeOf<OptionalValidatorSpec<string>>()
expectTypeOf(validator({ devDefault: undefined })).toEqualTypeOf<
RequiredValidatorSpec<string>
>()
expectTypeOf(validator({ default: undefined })).toEqualTypeOf<OptionalValidatorSpec<string>>()
expectTypeOf(validator({ devDefault: undefined })).toEqualTypeOf<
RequiredValidatorSpec<string>
>()
expectTypeOf(validator({ devDefault: 'foo' })).toEqualTypeOf<RequiredValidatorSpec<string>>()
expectTypeOf(validator<'foo'>({ devDefault: 'foo' })).toEqualTypeOf<
RequiredValidatorSpec<'foo'>
>()
expectTypeOf(validator({ default: 'foo', devDefault: 'foo' })).toEqualTypeOf<
RequiredValidatorSpec<string>
>()
expectTypeOf(validator<'foo' | 'bar'>({ default: 'foo', devDefault: 'bar' })).toEqualTypeOf<
RequiredValidatorSpec<'foo' | 'bar'>
>()
expectTypeOf(
validator<'foo' | 'bar'>({ choices: ['foo', 'bar'], devDefault: 'bar' }),
).toEqualTypeOf<RequiredValidatorSpec<'foo' | 'bar'>>()
})
test('structured data validator', () => {
const validator = makeMarkupValidator(() => ({}))
expectTypeOf(validator()).toEqualTypeOf<RequiredValidatorSpec<any>>()
expectTypeOf(validator({ default: {} as any })).toEqualTypeOf<RequiredValidatorSpec<any>>()
expectTypeOf(validator({ default: undefined })).toEqualTypeOf<OptionalValidatorSpec<any>>()
expectTypeOf(validator({ default: undefined, choices: [{ foo: 'bar' }] })).toEqualTypeOf<
OptionalValidatorSpec<{ foo: 'bar' }>
>()
expectTypeOf(validator({ devDefault: undefined })).toEqualTypeOf<RequiredValidatorSpec<any>>()
expectTypeOf(validator({ devDefault: { foo: 'bar' } })).toEqualTypeOf<
RequiredValidatorSpec<{ foo: string }>
>()
expectTypeOf(validator<{ foo: 'bar' }>()).toEqualTypeOf<RequiredValidatorSpec<{ foo: 'bar' }>>()
expectTypeOf(
validator({
default: {
hello: 'world',
},
}),
).toEqualTypeOf<
RequiredValidatorSpec<{
hello: string
}>
>()
expectTypeOf(
validator<{ hello: 'world' }>({
default: {
hello: 'world',
},
}),
).toEqualTypeOf<
RequiredValidatorSpec<{
hello: 'world'
}>
>()
expectTypeOf(validator<{ hello: string }>()).toEqualTypeOf<
RequiredValidatorSpec<{
hello: string
}>
>()
expectTypeOf(
validator({
choices: [
{ hello: 'world', option: false },
{ hello: 'world', option: true },
],
}),
).toEqualTypeOf<
RequiredValidatorSpec<{
hello: string
option: boolean
}>
>()
expectTypeOf(
validator({
choices: [
{ hello: 'world', option: false },
{ hello: 'world', option: true },
],
default: { hello: 'world', option: false },
}),
).toEqualTypeOf<
RequiredValidatorSpec<{
hello: string
option: boolean
}>
>()
})
})
test('cleanEnv', () => {
const env = {
STR: 'FOO',
STR_OPT: undefined,
STR_CHOICES: 'foo',
STR_REQ: 'BAR',
STR_DEFAULT_CHOICES: 'bar',
BOOL: 'false',
BOOL_OPT: undefined,
BOOL_DEFAULT: undefined,
NUM: '34',
NUM_DEFAULT_CHOICES: '3',
JSON_ANY: JSON.stringify(true),
JSON_REQ_ANY: JSON.stringify('Foo bar'),
JSON_DEFAULT: JSON.stringify({ foo: 'bar' }),
JSON_DEFAULT_OPT: undefined,
}
const specs = {
STR: str(),
STR_OPT: str({ default: undefined }),
STR_CHOICES: str({ choices: ['foo', 'bar'] }),
STR_REQ: str({ default: 'foo' }),
STR_DEFAULT_CHOICES: str({ default: 'foo', choices: ['foo', 'bar'] }),
BOOL: bool(),
BOOL_OPT: bool({ default: undefined }),
BOOL_DEFAULT: bool({
default: false,
}),
NUM: num(),
NUM_DEFAULT_CHOICES: num({ default: 1, choices: [1, 2, 3] }),
JSON_ANY: json(),
JSON_REQ_ANY: json({ default: {} as any }),
JSON_DEFAULT: json({ default: { foo: 'bar' } }),
JSON_DEV_DEFAULT: json({ devDefault: { foo: 'bar' } }),
JSON_DEFAULT_OPT: json<{ foo: 'bar' }>({ default: undefined }),
}
interface TestedCleanedEnv {
readonly STR: string
readonly STR_OPT?: string
readonly STR_CHOICES: 'foo' | 'bar'
readonly STR_REQ: string
readonly STR_DEFAULT_CHOICES: 'foo' | 'bar'
readonly BOOL: boolean
readonly BOOL_OPT?: boolean
readonly BOOL_DEFAULT: boolean
readonly NUM: number
readonly NUM_DEFAULT_CHOICES: 1 | 2 | 3
readonly JSON_ANY: any
readonly JSON_REQ_ANY: any
readonly JSON_DEFAULT: { foo: string }
readonly JSON_DEV_DEFAULT: { foo: string }
readonly JSON_DEFAULT_OPT?: { foo: string }
}
expectTypeOf(cleanEnv(env, specs)).toMatchTypeOf<TestedCleanedEnv>()
// Should also work when specs inlined
expectTypeOf(
cleanEnv(env, {
STR: str(),
STR_OPT: str({ default: undefined }),
STR_CHOICES: str({ choices: ['foo', 'bar'] }),
STR_REQ: str({ default: 'foo' }),
STR_DEFAULT_CHOICES: str({ default: 'foo', choices: ['foo', 'bar'] }),
BOOL: bool(),
BOOL_OPT: bool({ default: undefined }),
BOOL_DEFAULT: bool({
default: false,
}),
NUM: num(),
NUM_DEFAULT_CHOICES: num({ default: 1, choices: [1, 2, 3] }),
JSON_ANY: json(),
JSON_REQ_ANY: json({ default: {} as any }),
JSON_DEFAULT: json({ default: { foo: 'bar' } }),
JSON_DEV_DEFAULT: json({ devDefault: { foo: 'bar' } }),
JSON_DEFAULT_OPT: json<{ foo: 'bar' }>({ default: undefined }),
}),
).toMatchTypeOf<TestedCleanedEnv>()
})

@jsamr jsamr force-pushed the jsamr/enhanced-typings branch 4 times, most recently from af877e1 to 1443777 Compare November 13, 2022 15:38
@jsamr jsamr marked this pull request as ready for review November 13, 2022 15:41
@jsamr

This comment was marked as resolved.

Sign in to view
@jsamr jsamr marked this pull request as draft November 14, 2022 21:25
@jsamr jsamr marked this pull request as ready for review November 14, 2022 21:37
@jsamr
Copy link
Contributor Author

jsamr commented Nov 14, 2022

@af should be good!

@af
Copy link
Owner

af commented Nov 19, 2022

Thanks for this and sorry for the delay in looking at it! Hoping to review and test it out this weekend 👍

af
af reviewed Nov 20, 2022
View reviewed changes
Copy link
Owner

@af af left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some initial feedback, mostly about the exposed API and docs.

First of all, the JSON inference is super nice! And I really appreciate that you tidied up a lot of the kludges I still had in place from migrating this thing to TS.

Some of the types are pretty complex so I'm still digesting them. Overall looking very good though!

src/core.ts Show resolved Hide resolved
src/core.ts Outdated Show resolved Hide resolved
README.md Outdated

- `makeBaseValidator<BaseT>` when you want the output to be narrowed-down to a subtype of `BaseT` (e.g. `str`).
- `makeExactValidator<T>` when you want the output to be widened to `T` (e.g. `bool`).
- `makeMarkupValidator` for input which can produce arbitrary output types (e.g. `json`).
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if it makes sense to expose makeMarkupValidator in the public API. What other use cases can you think of for it? If we keep it private for now we can always expose it later. Given the domain of "parsing env var strings" I think json() would be the dominant case

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking query parameters, e.g. "option1=true&option2=false&token=XXX"; or POSIX like flags e.g. -b -c --long-flag=/bin/xxx, /etc/fstab, XML, YAML... JSON5; honestly I don't know much about backend developers practices , but I had imagined exposing this one would give plenty of flexibility for consumers of this library, freeing them from the hassle of typings these validators correctly.

PS1: didn't test but I think we should also forbid specs with choices in markup validators, correct?
PS2: I went back and forth between naming this validator markupValidator and structuredValidator ; English is not my native language and if you have better naming ideas I'm all for it

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think structuredValidator would be a better name for sure, when most people hear "markup" they probably think of a specific type of format like HTML/XML/etc.

Also re choices, a nice side effect of removing structuredValidator from the public API is we don't have to worry about that interaction :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also re choices, a nice side effect of removing structuredValidator from the public API is we don't have to worry about that interaction :)

Well, with one important exception: json validator. In any case, I did implement this specific logic here:
328e056

Yet, I'm happy to not export makeStructuredValidator if you don't want to.

README.md Outdated Show resolved Hide resolved
README.md Outdated
depending on your use case:

- `makeBaseValidator<BaseT>` when you want the output to be narrowed-down to a subtype of `BaseT` (e.g. `str`).
- `makeExactValidator<T>` when you want the output to be widened to `T` (e.g. `bool`).
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could probably use examples of each case (makeValidator vs makeExactValidator) here as I don't think it'll be clear to most readers without diving into the source

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me know if that's enough: 9727612

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't find the comment about formatting again; but fixed it here: ce1007f

src/core.ts Outdated Show resolved Hide resolved
@jsamr
Copy link
Contributor Author

jsamr commented Nov 20, 2022 •
edited

First of all, the JSON inference is super nice! And I really appreciate that you tidied up a lot of the kludges I still had in place from migrating this thing to TS.

Some of the types are pretty complex so I'm still digesting them. Overall looking very good though!

Hey @af ! It's cool if this PR proves useful. Although it was hard at times, I had a lot of fun with it. I don't know if it will help, but perhaps a trick to read those validator types is following:

// Such validator only works for subtypes of BaseT.
export interface BaseValidator<BaseT> {
  // These overloads enable nuanced type inferences for optimal DX
  // This will prevent specifying "default" alone from narrowing down output type
  (spec: RequiredChoicelessSpecWithType<BaseT>): RequiredValidatorSpec<BaseT>
  <T extends BaseT>(spec?: RequiredSpec<T>): RequiredValidatorSpec<T>
  <T extends BaseT>(spec: OptionalSpec<T>): OptionalValidatorSpec<T>
}

Think of it as function overloads. For instance, this:

export const num = makeValidator<number>((input: string) => {
  const coerced = parseFloat(input)
  if (Number.isNaN(coerced)) throw new EnvError(`Invalid number input: "${input}"`)
  return coerced
})

is strictly equivalent to this:

export function num(spec: RequiredChoicelessSpecWithType<number>): ValidatorSpec<number>
export function num<T extends number>(spec?: RequiredSpec<T>): RequiredValidatorSpec<T>
export function num<T extends number>(spec: OptionalSpec<T>): OptionalValidatorSpec<T>
export function num(spec?: Spec<number>): ValidatorSpec<number> {
  return internalMakeValidator((input: string) => {
    const coerced = parseFloat(input)
    if (Number.isNaN(coerced)) throw new EnvError(`Invalid number input: "${input}"`)
    return coerced
  })
}

In the call site, TypeScript will simply try each of these signatures, starting at the topmost, and stop at the one matching the combination of type parameters and arguments. E.g.:

const MY_NUM_VAR = num<1|2|3>();

In this instance, TypeScript picks the second overload from the top because:

  • There is a type parameter;
  • There is exactly zero argument.

@jsamr jsamr requested a review from af November 21, 2022 11:30
af
af requested changes Nov 24, 2022
View reviewed changes
Copy link
Owner

@af af left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few requested changes but looking good. Almost ready to merge!

src/validators.ts Outdated Show resolved Hide resolved
tests/basics.test.ts Show resolved Hide resolved
tests/types.test.ts Show resolved Hide resolved
src/types.ts Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@jsamr jsamr requested a review from af November 24, 2022 11:17
af
af approved these changes Nov 27, 2022
View reviewed changes
Copy link
Owner

@af af left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the revisions! Latest CI run failed but once that's resolved I'll merge 👍 :shipit:

tests/types.test.ts Show resolved Hide resolved
@jsamr
Copy link
Contributor Author

jsamr commented Nov 27, 2022

@af Should be good now!

@af af merged commit e06dc88 into af:main Nov 28, 2022
@jsamr
Copy link
Contributor Author

jsamr commented Dec 3, 2022

@af Please excuse the disturbance; I have a small request: would you be OK to release an alpha/beta with these changes? We use this package extensively in multiple projects, so I'll be able to beta-test this with reasonable coverage.

@af
Copy link
Owner

af commented Dec 3, 2022

Thanks for the nudge, I'd meant to do that but it got lost in the shuffle. Just published 8.0.0-alpha.1, let me know how it goes!

@jsamr jsamr mentioned this pull request Dec 4, 2022
Merged
@jsamr
Copy link
Contributor Author

jsamr commented Dec 8, 2022 •
edited

@af Happy to report that we've been using alpha.2 version in our 6 projects, and it's been all smooth!

@af
Copy link
Owner

af commented Dec 9, 2022

Great to hear, thanks! I will take it for a spin on a few of my projects as well, then after some long-overdue docs updates I'll push out a beta unless anything comes up

@jsamr jsamr mentioned this pull request Jan 5, 2023
Closed
@af af mentioned this pull request Jan 10, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@af af af approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@jsamr @af