assertion

Full identifier: https://w3id.org/np/RAPwHYQQtXh6p3DQQ066TmpKOBMIWkerAYv-chCViAqC0#assertion

Minted in Nanopublication

 CoSMO Semantic Post SemanticPost comment approve/disapprove edit as derived nanopublication

This is the identifier for the assertion of this nanopublication. https://w3id.org/np/RAPwHYQQtX...#assertion this assertion http://www.w3.org/2000/01/rdf-schema#comment comment (this is a literal) " Merging models trained for long with WIDEN When models were trained on a lot of data they diverged further from the baseline (e.g. in continual pretraining for additional languages), current merging methods underperform in this setting https://alphaxiv.org/pdf/2408.03092 @AlibabaGroup https://twitter.com/LChoshen/status/1823002789217493392/photo/1 How do you do that? Let's assume we update a matrix with a few models. Pick a pretrained model and consider the rest of the models as diff from it (task vectors) Normalize the row of each model, separating the normalization factor (magnitude) and direction (row) Now we weigh every row by how much it changed (higher = better) and average all together + some trick to sometimes keep the original weight so weights might not sum to 1. You can see how this follows recent findings about direction and size (e.g. https://x.com/prateeky2806/status/1727589818618523783) While the results in "just" merging are not changing that much, merging with a continually trained model (Sailor) that added many languages look quite good! https://twitter.com/LChoshen/status/1823002796259791276/photo/1 Criticism (@askalphaxiv didn't upload comment): There is a vast overclaiming calling Sailor a different pretrained model. Quite complex, hard to know if it will generalize and they only show a specific model. " .
This is the identifier for the assertion of this nanopublication. https://w3id.org/np/RAPwHYQQtX...#assertion this assertion https://schema.org/keywords keywords (this is a literal) "Sailor" .
This is the identifier for the assertion of this nanopublication. https://w3id.org/np/RAPwHYQQtX...#assertion this assertion https://schema.org/keywords keywords (this is a literal) "WIDEN" .
This is the identifier for the assertion of this nanopublication. https://w3id.org/np/RAPwHYQQtX...#assertion this assertion https://schema.org/keywords keywords (this is a literal) "large-language-models" .
This is the identifier for the assertion of this nanopublication. https://w3id.org/np/RAPwHYQQtX...#assertion this assertion https://schema.org/keywords keywords (this is a literal) "model-merging" .
This is the identifier for the assertion of this nanopublication. https://w3id.org/np/RAPwHYQQtX...#assertion this assertion https://schema.org/keywords keywords (this is a literal) "weight-disentanglement" .
This is the identifier for the assertion of this nanopublication. https://w3id.org/np/RAPwHYQQtX...#assertion The assertion above http://www.w3.org/ns/prov#wasGeneratedBy wasGeneratedBy This is a local identifier minted within the nanopublication. https://w3id.org/np/RAPwHYQQtX...#activity activity .
This is the identifier for this whole nanopublication. https://w3id.org/np/RAPwHYQQtX... This nanopublication date and time when the nanopublication was created http://purl.org/dc/terms/created was created on (this is a literal) "2024-09-03T21:16:16.131Z" .
This is a local identifier minted within the nanopublication. https://w3id.org/np/RAPwHYQQtX...#sig sig http://purl.org/nanopub/x/hasSignatureTarget has as target This is the identifier for this whole nanopublication. https://w3id.org/np/RAPwHYQQtX... this nanopublication .
This is a local identifier minted within the nanopublication. https://w3id.org/np/RAPwHYQQtX...#sig sig http://purl.org/nanopub/x/hasAlgorithm has the algorithm (this is a literal) "RSA" .

Sensenets, 3 Sep 2024, 21:16:16 UTC

References

Nanopublication Part Subject Predicate Object Published By Published On
links a nanopublication to its provenance http://www.nanopub.org/nschema#hasProvenance provenance
assertion
Sensenets
2024-09-03T21:16:16.131Z
links a nanopublication to its provenance http://www.nanopub.org/nschema#hasProvenance provenance
assertion
Sensenets
2024-09-03T21:16:16.131Z
links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion
assertion
Sensenets
2024-09-03T21:16:16.131Z
links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion
assertion
Merging models trained for long with WIDEN When models were trained on a lot of data they diverged further from the baseline (e.g. in continual pretraining for additional languages), current merging methods underperform in this setting https://alphaxiv.org/pdf/2408.03092 @AlibabaGroup https://twitter.com/LChoshen/status/1823002789217493392/photo/1 How do you do that? Let's assume we update a matrix with a few models. Pick a pretrained model and consider the rest of the models as diff from it (task vectors) Normalize the row of each model, separating the normalization factor (magnitude) and direction (row) Now we weigh every row by how much it changed (higher = better) and average all together + some trick to sometimes keep the original weight so weights might not sum to 1. You can see how this follows recent findings about direction and size (e.g. https://x.com/prateeky2806/status/1727589818618523783) While the results in "just" merging are not changing that much, merging with a continually trained model (Sailor) that added many languages look quite good! https://twitter.com/LChoshen/status/1823002796259791276/photo/1 Criticism (@askalphaxiv didn't upload comment): There is a vast overclaiming calling Sailor a different pretrained model. Quite complex, hard to know if it will generalize and they only show a specific model.
Sensenets
2024-09-03T21:16:16.131Z
links a nanopublication to its provenance http://www.nanopub.org/nschema#hasProvenance provenance
assertion
Sensenets
2024-09-03T21:16:16.131Z
links a nanopublication to its provenance http://www.nanopub.org/nschema#hasProvenance provenance
assertion
Sensenets
2024-09-03T21:16:16.131Z
links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion
assertion
Sensenets
2024-09-03T21:16:16.131Z
links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion
assertion
Sensenets
2024-09-03T21:16:16.131Z
links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion
assertion
Sensenets
2024-09-03T21:16:16.131Z
links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion
assertion
Sensenets
2024-09-03T21:16:16.131Z

Show All

Raw

TriG(txt), JSON-LD(txt), N-Quads(txt), XML(txt)