Bash: Is there a way to easily break a large array into smaller ones?

Question

I was wondering if there is an easy way in bash to break up a large array into several smaller ones. I currently am thinking something like this:

for ((i = 0; i<= (bigArrayLength/2); i++))  do   
bigArray[i] = smallArray[i]  
done

for ((i = (bigArrayLength/2); i <=bigArrayLength; i++))  
do  
bigArray[i] = secondSmallArray[i]  
done

But there has to be a better way to go about it. Any suggestions? Thanks!

Jobie

Answer 1

If you have bash version 3.2 or higher, you can do it using the new "subelement" syntax (${bigArray[@]:index_of_first_element:element_count}), but be careful — if the element values have any spaces in them, this could break horribly without preprocessing.

So the idea is along the lines of:

cnt="${#bigArray[@]}"
let cnt1="$cnt/2"
let cnt2="$cnt - $cnt1 - 1"
# this way we remove the rounding error if the count was odd and account for 0-based indexing

smallArray=( "${bigArray[@]:0:$cnt1}" )
secondSmallArray=( "${bigArray[@]:$cnt1:$cnt2}" )

Braque

You've flattened the arrays. You need smallArray=("${bigArray[@]:0:$cnt1}") and it will be OK with spaces in the elements. Also, slicing is supported in Bash 3.2 as well as 4. – Hermosillo

why the -1 thought? you are losing the last element – Mohsen

Answer 2

I appreciate lynxlynxlynx's answer here. They gave me enough to work with to get to an approach that worked for me. My large array use case involves an array size that is not known in advance, so I want to deal with the array in chunks of limited size.

Here is a concrete example with a sample "large" array that's isn't very large.

ARRAY=(a b c d e f g h i j k l m n o p q r s t u v w x y z)
CHUNK_SIZE=5

index=0
while [[ $index -lt ${#ARRAY[@]} ]]; do
    echo "${ARRAY[@]:$index:$CHUNK_SIZE}"

    index=$((index + CHUNK_SIZE))
done

The output from running this is:

a b c d e
f g h i j
k l m n o
p q r s t
u v w x y
z

In my real use, my array can contain tens of thousands of elements and my CHUNK_SIZE is 5000.

Weber