I have a table of 65536 2-byte values that I'm trying to store using PROGMEM in an Arduino Mega. I understand that there is a limitation of 32768 values per array, so I have split my table into several arrays as follows:
const PROGMEM uint16_t data1[] = {0x0001,0x0002....}; //16383 values in this array
const PROGMEM uint16_t data2[] = {0x0001,0x0002....}; //16383 values in this array
const PROGMEM uint16_t data3[] = {0x0001,0x0002....}; //16383 values in this array
const PROGMEM uint16_t data4[] = {0x0001,0x0002....}; //16383 values in this array
const PROGMEM uint16_t data5[] = {0x0001,0x0002....}; //4 values in this array
I am retrieving the values as follows:
value = pgm_read_word(&data1[index]);
value = pgm_read_word(&data2[index]);
//etc.
This seems to compile and run if I only have one of the 16383 value arrays present (others commented out). If I include more than one of the large arrays, the program will compile and the program storage spaces looks correct (using about 60% of program storage space including my code and the PROGMEM data), but the program will not run.
Is there some other data storage limit in the program space that I'm not aware of? Any ideas on how to get this running?
Memory addresses are normally 16bits, so to access memory above this limit, you need to use pgm_read_word_far and pgm_get_far_address.
WARNING: if you put these arrays at the top of your program, the rest of your program will be placed in the "far" part of the memory and everything will run slower because the µc will have to spend more time retrieving values from 24bits memory addresses, etc... (can't explain in detail because I dont know exactly what's happening)
Consequently, place those arrays and what uses them, at the end of your program. The less code in the far memory, the better.
HERE is an example code, which does not account my warning, but demonstrates the use of the far memory.
It looks as though if some of my data is in the "far" space and some is not, I'm still safe to use pgm_get_far_address and pgm_read_word_far to access all of it?
Still having issues. To back up a bit, I do have the arrays at the top of the program in #include header file. Based on your previous replies, it would seem that the program should still run correctly (although maybe slower since the program code is now in "far" memory) until I tried to access the arrays that were in "far" memory. However, the program did/does not seem to execute at all.
I have modified how I am trying to access the arrays as follows:
value = pgm_read_word_far(pgm_get_far_address(data1)+index);
value = pgm_read_word_far(pgm_get_far_address(data2)+index-16383);
//etc.
However, I'm still getting the same result - the program won't even begin to execute.
Is there a way to assign where in memory these arrays are stored so that I can keep my program code in "near" memory and have just the arrays in "far" memory?
After some more playing around, I have narrowed the problem down to the CAN library that I am using with an attached CAN shield. The library can be found here:
I can include the library with no problem, but as soon as I use the CAN.begin() function in my setup function (while also using the large PROGMEM arrays), the program will not execute.
#include <avr/pgmspace.h>
#include <SPI.h>
#include <CAN.h>
const PROGMEM uint16_t data1[] = {big array};
const PROGMEM uint16_t data2[] = {big array};
const PROGMEM uint16_t data3[] = {big array};
const PROGMEM uint16_t data4[] = {big array};
unsigned int k = 0;
CanMessage RXmessage;
CanMessage TXmessage;
unsigned char TXdata[8] = {0,0,0,0,0,0,0,0};
unsigned char RXdata[8] = {0,0,0,0,0,0,0,0};
void setup() {
Serial.println("attempting to start CAN");
CAN.begin(CAN_SPEED_500000); //if I comment out this line and the line below, it works
CAN.setMode (CAN_MODE_NORMAL);
Serial.begin(57600);
Serial.println("CAN started");
}
void loop() {
while(k<16383*2){
Serial.println(pgm_read_word_far(pgm_get_far_address(data1)+k),HEX);
Serial.println(pgm_read_word_far(pgm_get_far_address(data2)+k),HEX);
Serial.println(pgm_read_word_far(pgm_get_far_address(data3)+k),HEX);
Serial.println(pgm_read_word_far(pgm_get_far_address(data4)+k),HEX);
Serial.println("I am running");
delay(1000);
k = k+2;
}
k = 0;
}
If I comment out the two lines shown above, then the program will run and I can access my arrays in far memory.
Is there a way that I can track down the issue that the library is having with far memory and correct it?
Some libraries need accurate timings to communicate with hardware. Those timings may be wrong if the code is ran in the far memory. I'm affraid it won't be easy to fix.
But I took a quick look at that library and found this in mcp2515.cpp:
/** For each increment of the BRP, our time quanta goes up this many
* nanoseconds (for our 16MHz crystal) */
#define TIME_QUANTUM_STEP 125
Try to change this number (maybe to 250). I'm really not sure it will help but worth trying.
Or, you could follow my advice about moving those arrays (and the code that uses them, of course) at the bottom of your code.
guix:
Some libraries need accurate timings to communicate with hardware. Those timings may be wrong if the code is ran in the far memory. I'm afraid it won't be easy to fix.
I would buy that if I was only having problems with CAN communication. However, I don't even get the very first Serial.println() in the code to work, which is before I attempt any SPI/CAN communication. I don't think the issue is that I can't communicate with the hardware, it's the the code doesn't seem to be executing at all.
guix:
Or, you could follow my advice about moving those arrays (and the code that uses them, of course) at the bottom of your code.
I tried this with no impact...maybe I'm not doing it correctly. I moved the array declarations just above the function that uses them, and put both at the bottom of my code. Same result.
Probably CAN used PROGMEM as well, and has its variables forced into high memory, but doesn't use the far access functions. There may be a way to force your data to be last, allowing other PROGMEM to be "near"...
From what I understand, this is telling avr-gcc to put the arrays into the .fini1 memory section, which will be after the program code (after return from main() or a call to exit()). I'm stealing that from here:
However, that link also says that .fini1 is "user definable". What does that mean exactly? Why .fini1 and not .fini2, etc? Or are those just placeholders? If I put something in .fini2 would be be before .fini1?
The compiler produces "relocatable" segments of code, and the linker figures out how to place these in real memory based on information in a "linker script" (contained in .../tools/avr/avr/lib/ldscripts/...)
All that "PROGMEM" really does is put the designated pieces of data in a section called .progmem.something.
The Linker script for an ATmega1280 is something like:
.text :
{
*(.vectors)
KEEP(*(.vectors))
/* For data that needs to reside in the lower 64k of progmem. */
*(.progmem.gcc*)
*(.progmem*)
/* For code that needs to reside in the lower 128k progmem. */
*(.lowtext)
*(.init0) /* Start here after reset. */
KEEP (*(.init0))
*(.init1)
KEEP (*(.init1))
:
*(.init9) /* Call main(). */
KEEP (*(.init9))
*(.text)
. = ALIGN(2);
*(.text.*)
. = ALIGN(2);
*(.fini9) /* _exit() starts here. */
KEEP (*(.fini9))
:
*(.fini0) /* Infinite loop after program termination. */
KEEP (*(.fini0))
_etext = . ;
} > text
.data ...
The PROGMEM stuff (.progmem) is intentionally loaded "early" in the memory so that it will be in the bottom 64k where it is easy to reach with pgm_get_byte() and similar. Which is just fine when there is a relatively small amount of PROGMEM data, but doesn't make sense at all if you KNOW that there is going to be more than 64k of PROGMEM. In that case, you want to put the data that is used by the generic libraries, that NEEDS it to be in the first 64k, in the early memory, and your large bulk data, that you know you'll need to use the _far() functions to access, after everything else. It would probably be nice if there were a section named ".farprogmem" or something that you could use, but there isn't, and the effort of adding it doesn't really seem justified as long as those .fini sections are there anyway...
