Is there a practical way of migrating from identity columns to hilo keys?

Question

I work with a database that depends heavily on identity columns. However as we have now moved all applications over to NHibernate I wanted to look into using HiLo as seems to be recommended with NHibernate. Are there any strategies to do this, or any common problems to watch out for?

Fabre

Answer 1

If this is a question about migrating an existing application to hilos which previously used auto ids, and has old data in it which needs to be migrated... then this would be my best bet (not tried it though! - comments welcome!) :

• Change your column types ids to bigints
• find out highest id value currently in any table.
• Set your 'next-high' value in the hilo source table to a value higher than you found in the IDs

If course this only addresses issues with the identity column, not anything else in your schema that might need to change if you are moving an app to NHibernate.

Coleencolella

Actually, you shouldn't be setting the next-hi value to a value higher than the one found in the IDs, see Stefan's answer below for a good approach. – Haldane

@legenden - care to explain why not? – Coleencolella

@UpTheCreek: It is too high, you loose a lot of ids. In the mapping file, you specify how many ids you can generate for a single value in next-hi. next-hi is therefore always smaller by some factor as the generated ids. – Mathematician

Ok - I defer to Stefan's answer in that case :) – Coleencolella

Hi, why about Change your column types ids to bigints ? – Agnail

Answer 2

You need to setup the table used by NH to create HiLo values correctly. Let Schema Creator create the table according to your mapping definitions, set the values according to the current state of the ids in your database.

I believe (you need to verify this) that values generated by hilo are calculated by:

hilo-id = high-value * max_lo + low-value

While the high-value is stored in the database, max_low defined in the mapping file and low-value calculated at runtime.

---

NHibernate also needs its own connection and transaction to determine and increment the high value. Therefore it does not work if the connection is provided by the application.

You can still use seqhilo, NH uses a database sequence to create next high-values and does not need a separate connection to do so. This is only available on databases which support sequences, like Oracle.

---

Correction:

Meanwhile, I had to implement it myself (before, it was just theory :-). So I come back to share the details.

The formula is:

next_hi = (highest_id / (maxLow + 1)) + 1

next_hi is the field in the database you need to update. highest_id is the highest ID found in your database. maxLow is the value you specified in the mapping file. No Idea why it is incremented by one. The division is an integer devision which truncates the decimal places.

Mathematician

Should you have a hilo table per entity table? – Accouplement

Answer 3

I wrote a script (based of Stephans answer) for fixing hilo values (on sql server) - it assumes that you have a hilo table like

CREATE TABLE [dbo].[HiloValues](
    [next_hi] [int] NULL,
    [Entity] [varchar](128) NOT NULL
)

And your table's identity columns are all called ID. Init the Entity table with the table names you want to generate the hilo values for. Running the script will generate a series of update statements like this one:

UPDATE hv 
SET next_hi = Transactions.ID/(10 + 1) + 1 
FROM HiloValues hv 
CROSS JOIN (SELECT ISNULL(Max(ID), 0) as id FROM Transactions) as Transactions
WHERE hv.entity = 'Transactions'

Here it is

DECLARE @scripts TABLE(Script VARCHAR(MAX))
DECLARE @max_lo VARCHAR(MAX) = '10';

INSERT INTO @scripts
SELECT '
UPDATE hv 
SET next_hi = ' + Entity + '.ID/(' + @max_lo + ' + 1) + 1 
FROM HiloValues hv 
CROSS JOIN (SELECT ISNULL(Max(ID), 0) as id FROM ' + entity + ') as ' + entity + '
WHERE hv.entity = ''' + entity + '''' as script 
FROM HiloValues WHERE Entity IN (SELECT  name from sys.tables)



DECLARE curs CURSOR FOR SELECT * FROM @scripts
DECLARE @script VARCHAR(MAX)

OPEN curs 
FETCH NEXT FROM curs INTO @script

WHILE @@FETCH_STATUS = 0
BEGIN
    PRINT @script --OR EXEC(@script)
    FETCH NEXT FROM curs INTO @script
END
CLOSE curs
DEALLOCATE curs

Accouplement

Answer 4

Here is a sample from a recent migration from the increment generator to the MultipleHiLoPerTableGenerator (e.g. a single table is used to store high values for every Entities).

My application is using Hibernate 3 + mapping files (.hbm.xml). My database is MySQL (innoDB + auto increment pk).

Step 1 : replace your generator settings in your .hbm files. Replace :

<generator class="increment" />

By

<generator class="org.hibernate.id.MultipleHiLoPerTableGenerator">
    <param name="table">hilo_values</param>
    <param name="primary_key_column">sequence_name</param>
    <param name="value_column">sequence_next_hi_value</param>
    <param name="max_lo">1000</param>
</generator>

Step 2 : create a new table to store the high values

CREATE TABLE IF NOT EXISTS `hilo_values` (
  `sequence_name` varchar(255) NOT NULL,
  `sequence_next_hi_value` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Step 3 : populate the initial high values according to existing data using the following piece of SQL. I assume here that the same max_lo value is used for every table.

INSERT INTO hilo_values SELECT TABLE_NAME,  ((AUTO_INCREMENT DIV (1000 + 1)) + 1) FROM information_schema.tables WHERE table_schema = 'yourdbname'

Alerion

Answer 5

Here's a script (MS SQL) that will fill HiLo(Name,Value) table with all next high numbers for all tables in current database:

declare tables cursor for

    select
        Table_Schema,
        Table_Name
    from
        information_schema.tables
    where
        Table_Schema = 'dbo'
        and
        Table_Type = 'BASE TABLE'
        and
        Table_Name <> 'HiLo'
        and
        right (Table_Name, 1) <> '_'

declare @table_schema varchar(255)
declare @table_name varchar(255)

truncate table HiLo

open tables
fetch next from tables into @table_schema, @table_name

while (@@fetch_status = 0)
begin
    declare @sql as nvarchar(max)
    declare @max_id as int

    set @sql = 'select @max_id = max(Id) from [' + @table_schema + '].[' + @table_name + ']'
    exec sp_executesql @sql, N'@max_id int output', @max_id output

    declare @max_low as int
    set @max_low = 1000

    declare @next_high as int
    set @next_high = isnull (@max_id / @max_low + 1, 0)

    --select @table_name, @max_id, @next_high
    insert into HiLo (Name, Value) values (@table_schema + '.' + @table_name, @next_high)

    fetch next from tables into @table_schema, @table_name
end

close tables
deallocate tables

select * from HiLo

Concubinage