lassence.net

Field notes from the AI frontier

Assign XP groups with hash functions

--- date: · theme: data · words: 434 ---

Introduction

In the case of statistical experiments, some hash functions that produce integers can be used for easily pseudo-randomly and deterministically assign users to groups.

The process works as follows:

  1. Hash a unique identifier of each user, for example their UID –> it will generate a large integer
  2. Optionally get the absolute value of the hash, because it can be negative
  3. Apply a modulo to the integer, corresponding to the number of experimentation groups that you want

Hash in Python

⚠️ Warning: in Python, using the built-in hash() function will produce different outputs across sessions if you don’t explicitly specify the same hash seed value every time.

# Set hash seed
%env PYTHONHASHSEED=0
# Import libraries
import pandas as pd

# Sample dataframe of UIDs
df = pd.DataFrame({'uid':[
    'd4212ed3-4399-4f5b-87ab-faec4f6558f8',
    'a9780c08-9714-4da2-a53c-6102d5f47cf7',
    '26c6853f-cae9-4a06-849d-7df46d744e5a',
    'efc1583b-946a-449f-a811-87d241e57bc4',
    '60d568e2-052b-4ca3-83da-34ee00ef5c6b'
]})
df
uid
0d4212ed3-4399-4f5b-87ab-faec4f6558f8
1a9780c08-9714-4da2-a53c-6102d5f47cf7
226c6853f-cae9-4a06-849d-7df46d744e5a
3efc1583b-946a-449f-a811-87d241e57bc4
460d568e2-052b-4ca3-83da-34ee00ef5c6b
# Step 1a: compute hash value for each user UID
df['uid_hash'] = df['uid'].apply(hash)
df
uiduid_hash
0d4212ed3-4399-4f5b-87ab-faec4f6558f88654918514890274169
1a9780c08-9714-4da2-a53c-6102d5f47cf71340609583511541598
226c6853f-cae9-4a06-849d-7df46d744e5a-1679195277781215170
3efc1583b-946a-449f-a811-87d241e57bc4-2786820932382908882
460d568e2-052b-4ca3-83da-34ee00ef5c6b3180348019046944236
# Step 1b: get the absolute value of the hash
df['abs_hash'] = df['uid_hash'].apply(abs)
df
uiduid_hashabs_hash
0d4212ed3-4399-4f5b-87ab-faec4f6558f886549185148902741698654918514890274169
1a9780c08-9714-4da2-a53c-6102d5f47cf713406095835115415981340609583511541598
226c6853f-cae9-4a06-849d-7df46d744e5a-16791952777812151701679195277781215170
3efc1583b-946a-449f-a811-87d241e57bc4-27868209323829088822786820932382908882
460d568e2-052b-4ca3-83da-34ee00ef5c6b31803480190469442363180348019046944236
# Step 3: apply a modulo, to reduce each hash
# In this example, we want 5 groups
groups = 5
df['user_group'] = df['abs_hash'].apply(lambda x: x % groups)
df
uiduid_hashabs_hashuser_group
0d4212ed3-4399-4f5b-87ab-faec4f6558f8865491851489027416986549185148902741694
1a9780c08-9714-4da2-a53c-6102d5f47cf7134060958351154159813406095835115415983
226c6853f-cae9-4a06-849d-7df46d744e5a-167919527778121517016791952777812151700
3efc1583b-946a-449f-a811-87d241e57bc4-278682093238290888227868209323829088822
460d568e2-052b-4ca3-83da-34ee00ef5c6b318034801904694423631803480190469442361

Hash in Redshift

Redshift provides several hash functions, with one algorithm named FNV outputing integers with FNV_HASH().

-- Redshift hash with FNV_HASH() function
SELECT
    uid,
    FNV_HASH(uid) AS uid_hash,
    ABS(FNV_HASH(uid)) AS abs_hash,
    MOD(ABS(FNV_HASH(uid)), 5) AS user_group
FROM users
uiduid_hashabs_hashuser_group
e5bc5b34-7591-440b-afe5-abd09ce9e7fc322780394103450642932278039410345064294
45537e99-fb26-4f91-bc05-54609a1aaea4-512213372145938410451221337214593841044
dd654604-c782-4bb7-b8b1-f4d29544901d-661798139063347998866179813906334799883
cdd58d6c-cc32-4d08-9bc2-5dbd9794a850-843138662605872678984313866260587267894
312fa481-ade1-4049-a0a2-1f8fd8127038-657519230477936499965751923047793649994
92a78700-1776-4bdd-9a30-3f1e2a5215ad-350290904911767227735029090491176722772
ff9bf307-46ed-4eaa-8768-56d9bd243638-389632612968512073338963261296851207333
df0feb24-46fb-4413-a6dd-bc720a8404e0-157415629774057223315741562977405722333
c88e885f-7e07-4164-bbef-8b3e06b01c03-219344578556954943821934457855695494383
e6840bd5-a7f8-4bca-b3c4-dd0d063bd805723221339819204227172322133981920422711

Hash in BigQuery

BigQuery provides a deterministic fingerprinting based on the FarmHash library with the FARM_FINGERPRINT() function

-- BigQuery provides FARM_FINGERPRINT() for hash
SELECT
    uid,
    FARM_FINGERPRINT(uid) AS uid_hash,
    ABS(FARM_FINGERPRINT(uid)) AS abs_hash,
    MOD(ABS(FARM_FINGERPRINT(uid)), 5) AS user_group
FROM users
uiduid_hashabs_hashuser_group
e5bc5b34-7591-440b-afe5-abd09ce9e7fc-778756422446693459077875642244669345900
45537e99-fb26-4f91-bc05-54609a1aaea4560831802508286274556083180250828627450
dd654604-c782-4bb7-b8b1-f4d29544901d593420908596864218259342090859686421822
cdd58d6c-cc32-4d08-9bc2-5dbd9794a850-6163264173159190326163264173159190322
312fa481-ade1-4049-a0a2-1f8fd8127038890610232042303538289061023204230353822
92a78700-1776-4bdd-9a30-3f1e2a5215ad-694450359384830038269445035938483003822
ff9bf307-46ed-4eaa-8768-56d9bd2436389411761598130389319411761598130389311
df0feb24-46fb-4413-a6dd-bc720a8404e0-701119865773005570170111986577300557011
c88e885f-7e07-4164-bbef-8b3e06b01c03-773588286615144180777358828661514418072
e6840bd5-a7f8-4bca-b3c4-dd0d063bd805784360124299256994078436012429925699400